google cloud platform: prototype ->production-> planet scale
TRANSCRIPT
Nir ChinskyHead of Google Cloud Platform, MEA
Google Cloud Platform 2
1 Billion Users
Data &Analytics
Infrastructure& Operations
Application Development
Confidential & ProprietaryGoogle Cloud Platform 5
Everything You Need To Build And ScaleComputeFrom virtual machines with proven price/performance advantages to a fully managed app development platform.Compute EngineApp EngineContainer EngineContainer RegistryCloud Functions
Storage and DatabasesScalable, resilient, high performance object storage and databases for your applications.Cloud StorageCloud BigtableCloud DatastoreCloud SQL
NetworkingState-of-the-art software-defined networking products on Google’s private fiber network.Cloud Virtual NetworkCloud Load BalancingCloud CDNCloud InterconnectCloud DNS
Management ToolsMonitoring, logging, and diagnostics and more, all a easy to use web management console or mobile app.Stackdriver OverviewMonitoringLoggingError ReportingDebuggerDeployment Manager & More
Big DataFully managed data warehousing, batch and stream processing, data exploration, Hadoop/Spark, and reliable messaging.BigQueryCloud DataflowCloud DataprocCloud DatalabCloud Pub/SubGenomics
Machine LearningFast, scalable, easy to use ML services. Use our pre-trained models or train custom models on your data.Cloud Machine Learning PlatformVision APISpeech APITranslate API
Developer ToolsDevelop and deploy your applications using our command-line interface and other developer tools.Cloud SDKDeployment ManagerCloud Source RepositoriesCloud EndpointsCloud Tools for Android StudioCloud Tools for IntelliJGoogle Plugin for EclipseCloud Test Lab
Identity & SecurityControl access and visibility to resources running on a platform protected by Google’s security model.Cloud IAMCloud Resource ManagerCloud Security ScannerCloud Platform Security Overview
Hitting the limits, early on...
The Anatomy of a Large-Scale Hypertextual Web Search Engine
1996, Sergey Brin and Lawrence PageComputer Science Department, Stanford University, Stanford, CA 94305
Building on Google’s infrastructure
2.8 million devices
activatedEvery Day (1.1 billion
devices)
10 billionhours
watchedEvery Month (100 hours
of new content uploaded every minute)
43 billionpages
crawledEvery Day
Challenges
11010101010010
1
1011101
011110010001101
Legacy Applications
Regulatory Environment
Changing View on Value of
Data
No Cloud Skills, Hard to
Recruit
Data Silos Everywhere
GOOGLE CONFIDENTIAL
We want to change what you can do.Not just where you can do it.
Select
IaaS
OS Setu
p
Containerization
Storage
Container Orchestrati
on
Networking
Monitoring
Logging
Testing
DevUser
Prototype
Production
Planet-scale
Original Launch Target Estimated Worst Case Actual Traffic
Cloud Datastore Transactions Per Second
50XActual Traffic
5XWorst Case
Estimate
1XTarget Traffic
Google Cloud Platform 17
The Journey to a Web-Scale Cloud
2nd Wave1st Wave 3rd Wave
Open Matters
Walking the Walk
Leading Open Source Communities
#1 Highest Engagement on Github #2Highest
Engagement on Github
Kubernetes Tensorflow
Source: Analyzing GitHub issues and comments with BigQuery
KubernetesApps
600+Security
Engineers
Security and Compliance
700+CVEs Discovered
and/or Fixed
160Academic ResearchPapers on Security
Google’s Data Research
2002
2004
2006
2008
2010
2012
2014
2016
GFS
MapReduce
TensorFlow
BigTable
Dremel
Colossus
Flume
Megastore
Spanner
Millwheel
PubSubF1
Google’s Data Products
2002
2004
2006
2008
2010
2012
2014
2016
ML
PubSub
DataFlow
DataStore
DataFlow
Cloud Storage
BigQuery
BigTable
DataProc
Cloud Storage
Data & Analytics
Insight Data
Programming
Resource provisionin
gPerformance tuning
Monitoring
Reliability
Deployment & configuration
Handling growing
scale
Utilization improvemen
ts
Typical Big Data Jobs
Big Data with GoogleFocus on insights.Not infrastructure.
Programming
Understanding
Dataproc: Fully managed Hadoop and Spark w/ industry-leading performance
BigQuery: Fully managed data warehouse for large-scale analytics
Dataflow: Real-time data pipelines, with open source SDK via Apache Beam
Big Data with Google
30
31
No DBA
NBD(no big deal)
“Right at the start of the partnership we were able to reduce time to insight from 96 hours to 30 minutes by using
BigQuery”
Gary SandersHead of Digital Analytics
Music for Everyone
75M+ Users2B+ Playlists30M+ Songs
Data is the center of the Spotify music experience
With GCP, data teams get big data insights in minutes versus hours
“From traditional batch processing to rock-solid event delivery to the nearly magical abilities of BigQuery, building on Google’s data
infrastructure provides us with a significant advantage where it matters the most.”
Nicholas HarteauVP of Engineering and Infrastructure
Analytics to Machine Learning
Cloud Vision API
Cloud Translate API Cloud
Natural Language API
Cloud Speech API
Ready to Use Machine Learning
Models
Cloud Machine Learning
DEVELOP - MODEL - TEST
Use Your Own Data To Train Models
Hypertune
Cloud Machine Learning
NoOps Machine Learning
Training at Scale
Cloud Datalab UI
Rapidly Accelerating Use of Deep Learning at Google
Number of directories containing model description files
2012 2013 2014 2015
1500
1000
500
0
Used across products:
Auto-organizing photo library
We will group your photos together,
recognize the important people, and
group them in collections
for you to view.
We just released this feature worldwide couple weeks ago.
Navigate your photos by place
We can understand the
objects and things in your
photos
PhotoNet by Google
DeepMind
Confidential & ProprietaryGoogle Cloud Platform 44
Cloud Vision APIInsight from images with our
powerful Cloud Vision API
FacesFaces, facial landmarks, emotions
OCRRead and extract text, with support for > 80 languages
Cloud Vision APICall API from anywhere, with support for embeddable images, and Google Cloud storage
LabelDetect entities from furniture to transportation
LogosIdentify product logos
Landmarks & Image PropertiesDetect landmarks & dominant color of image
Safe SearchDetect explicit content - adult, violent, medical and spoof
API Usage: Detect Objects in an Image
Image Detected
ItemsVision API
Upload ImagesCloud/On premise
Process the response
Call the REST API1 2 3
Digital Transformation = APIs
