a sentient network - how high-velocity data and machine learning will shape the future of...
TRANSCRIPT
A Sentient NetworkHow High-velocity Data and Machine Learning will Shape the Future of the Communication Services
OPNFV Summit, Burlingame CA, November 9-12, 2015
Wenjing ChuDistinguished EngineerDell ResearchMember of the Board and TSC of OPNFV
“The more real-time and granular we can get, the more responsive, and more competitive, we can be.”
Peter Levine | Andreessen Horowitz
A Sentient Network
1Elastic on-demand capacityOpen software architecture promises flexible elastic capacity that can be rapidly provisioned and dynamically managed
Data-driven operation automationVirtualization unleashes the latent value in the real-time data to optimize resource allocation and assure SLA
Scalable infrastructureStandard open architecture infrastructure delivers capacity, cost efficiency, and right-sized reliability
2
3Self-learning security and privacySelf-learning algorithms from real-time data delivers ultimate security and privacy at the same time
4Machine intelligent user servicesAdvances in Machine Learning promise continuous improvements in user experience
5
New Paradigm Shift in Infrastructure: NFV/SDNDomain specialized software on standard hardware, delivered from the cloud- Dramatically cuts CapEx & OpEx- Enhances service velocity- Enables Big Data driven business model
High-velocity Cloud Empowers Business Transformation
Mobile' Infrastructure
Content' Distribution
Edge Computing
High'Velocity'Cloud
Packet Velocity• 100X more performance• 50X more customers
Service Velocity• Deploy services in minutes vs months• Empower new, innovative business
models
Data Analytics Velocity• Sub-second real-time streaming
analytics• Sentient intelligence
High-velocity Data with Machine Learning
Telemetry, IoT
sensors, System logs,
Monitors, Mobile
devices …
Transmission of data in streams
Transformation
Learning in real-
time
Action on
intelligence
“Meta Dimensionality” of Data
Gigabyte, Terabyte, Petabyte, Exebyte, Zettabyte, Yottabyte
uSec
, mSe
c, S
ec, m
in, d
ays,
mon
ths,
year
s…
SizeTi
me
Automatically Adjust Resources to Maintain SLA
Systems can respond to usage spikes in real-time, to reallocate resources and maintain SLAs.
Continuous Resource Optimization by Reinforcement Learning
! Modeled as a Markov Decision Process
! Learning probability distribution by Bayesian inference
! Q-Learning, Deep Q-Network
! Consensus optimization
Wikipedia: MDP
Classification by Concept Adapting Decision Tree
! Rules programming is labor intensive, error prone, static
! Let algorithm learns a DT (or a forest) on its own
! Concept adaptability: incorporate new, forget old
Packets > 10
yes no
Protocol=http
Packets > 10
noyesBytes > 60k
yes no
Protocol=ftp
Data stream
Data stream
Uncovering Unusual Hidden Activity by Monitoring Entropy
! Entropy in a moving time window captures the normal humming of the system
! Out of ordinary move of entropy plus context suggest attack vs. flash crowd
Clustering Users based on Behavior Patterns
! Non-parametric model can be used for latent features, overlapping clusters and infinite data
! Eg Dirichlet process, Gaussian process
! A cluster of ‘users’ of abnormal behavior are suspects
! Mining telco CDR’s to evaluate risks from customer churn
! Combining location and real-time system info to pinpoint quality issues
! Machine learning algorithm offers more precision
Proactive Customer Support and Retention
The peaks indicate areas of highest risk with more precision than traditional linear regression (the dotted line).
Creative commons http://scicomp.stackexchange.com/
Collaborative Learning by Sensing User Mood
Facial expressions
Pulse rate
Skin conductivity
Brain computer interface (BCI)
Voice pitch
Remote UX metrics
Media audience response
Improve MOOC, CBT
VR/AR styleUI
20
“How is Seamless Mobility powered by High Velocity Cloud?” Seamless Mobility by Contextual Learning
Live machine learning algorithms ensure quality, security and seamless mobility.
High-velocity Cloud
High-velocity Analytics
Differential Privacy in Big Data and Machine Learning
! Anonymization is not enough
! Differential Privacy (!-DP) provides a formal guarantee & a mechanism for tradeoff
! DP may also help avoid False Discovery
Dr.Katrina Ligget, CalTech
Computing on and Learning from Encrypted Data
transformed+queryplain+query+
under+passive+attack
Applicationdecrypted+results
encrypted+results
DB+server
encrypted+DBProxy
SecretSecret
computation+on+encrypted+data+≈+
regular+computation
! Stores+schema++and+master+key
! No+query+execution
trusted+client?side! Data loss is prevalent
everywhere you look! Data privacy
responsibility is unclear! Practical system can be
deployed with strong encryption without the risk of key disclosure
! Different algorithm for different computation
Dr. Laruca Popa, UC Berkeley
So, Any Takeaways for OPNFV ?
• Collect data• Put data in an open format• Consider privacy and security on day one• Don’t tie data to a specific implementation of a specific design• Must consider the time dimension of data, e.g. TSDB, streaming