gcp - gce, cloud sql, cloud storage, bigquery basic training

MiTAC MiCloud - Google Cloud Platform Partner @ APAC2014Q1 GCP Introduction

Google Cloud Platform - Starter Guide

Start from Create Google Cloud Platform Project

Start from Google Cloud Project

Create Project

http://cloud.google.com/console

Enable Billing

Related SDK / Tool Installation

Google Cloud Storage - Object storage service, without limit and global deployed

watch videos, screen casts, and presentations that walk through how to use Google

Cloud Storage.

VIDEOS

download sample applications, read howto

guides, and learn how to use Google Cloud Storage with

other Google products.

SAMPLES

ask questions, discuss solutions, and join our vibrant community of

developers.

COMMUNITY

Google Cloud Storage

GCS - Features

● High Capacity and Scalability● Strong Data Consistency● Google Cloud Console Projects● Bucket Locations● REST APIS● OAuth 2.0 Authentication● Authenticated Browser Downloads● Google Account Support for Sharing

Getting Start

Lab1 - Build a Storage Web Site

Step 1: Create your siteStep 2: Upload your site & setup permissionStep 3: Make your bucket serve as a web siteStep 4: Setup Domain CNAME mapping

Step 1: Create your site

https://jetstrap.com

Step 2: Upload & Permission

Step 3: Serve as a web site

$ gsutil web set -m index.html -e 404.html gs://gsweb.micloud.tw

Step 4: Setup Domain CNAME mapping

Step 4: Setup CNAME mapping

Quick Start● Features: https://cloud.google.com/products/cloud-

storage#features● Case Studies: https://cloud.google.com/products/cloud-

storage#case-studies● Pricing: https://cloud.google.com/products/cloud-storage#pricing● Documentation: https://cloud.google.com/products/cloud-

storage#documentation

Google Cloud SQL - Full managed MySQL database instance

Google Cloud SQL

Features

● Familiar Infrastructure● Flexible Charging● Security, Availability, Durability● EU and US Data Centers● Easier Migration; No Lock-in● Managed by Google

Getting Start

Permission Control

Request a IP for management

Setup connect from IP address

● Create a Bucket● Create a CloudSQL

Lab 2 - Connect to CloudSQL

Create a Bucket

$ gcloud auth login

$ gcloud config set project [my-project-id]

$ gsutil mb gs://[my-bucket]

$ gsutil ls gs://[my-bucket]

Sample Database

http://www.mysqltutorial.org/mysql-sample-database.aspx

Create a CloudSQL

$ gcloud sql -h

$ gcloud sql instances create [inst-name]

$ gcloud sql instances list

$ gcloud sql instances import instance-name --uri \

gs://your-bucket/sql-dump-file.gz

$ gcloud sql instances patch --assign-ip [inst-name]

$ gcloud sql instances get [inst-name]

Create a CloudSQL

$ gcloud sql instances set-root-password test-sql \

-p [password]

$ gcloud sql instances patch test-sql \

--authorized-networks=[your-ip-address]

Connect Cloud SQL - MySQL Workbench

http://www.mysql.com/products/workbench/

Advance Connect - SSH Tunnel

● Configuring Replication with Google Cloud SQL: https://developers.google.com/cloud-sql/docs/replication

Others

Quick Start● Features: https://cloud.google.com/products/cloud-sql#features● Case Studies: https://cloud.google.com/products/cloud-sql#case-

studies● Pricing: https://cloud.google.com/products/cloud-sql#pricing● Documentation: https://cloud.google.com/products/cloud-

sql#documentation

Google BigQuery - Big data with SQL like query feature, but fast...

Google BigQuery

BigQuery Features

● TB level data analysis● Fast mining response● SQL like query language● Multi-dataset interactive

support● Cheap and pay by use● Offline job support

Getting Start

BigQuery structure

● Project● Dataset● Table● Job

Import Wizard

Lab3 - Load CSV to BigQuerygsutil cp [source] gs://[bucket-name]# gsutil cp ~/Desktop/log.csv gs://your-bucket/Copying file:///Users/simonsu/Desktop/log.csv [Content-Type=text/csv]...Uploading: 4.59 MB/36.76 MB

bq load [project].[dataset] gs://[bucket]/[csv path] [schema]# bq load project.dataset gs://your-bucket/log.csv IP:STRING,DNS:STRING,TS:STRING,URL:STRING

Waiting on bqjob_rf4f3f1d9e2366a6_00000142c1bdd36f_1 ... (24s) Current status: DONE

/* Ref: https://developers.google.com/apps-script/advanced/bigquery */var request = { query: 'SELECT TOP(word, 30) AS word, COUNT(*) AS word_count ' + 'FROM publicdata:samples.shakespeare WHERE LENGTH(word) > 10;' };var queryResults = BigQuery.Jobs.query(request, projectId);var jobId = queryResults.jobReference.jobId;queryResults = BigQuery.Jobs.getQueryResults(projectId, jobId);var rows = queryResults.rows;while (queryResults.pageToken) { queryResults = BigQuery.Jobs.getQueryResults(projectId, jobId, { pageToken: queryResults.pageToken }); rows = rows.concat(queryResults.rows);}

Apps Script Code

Others● Load JSON to bigquery

https://sites.google.com/a/mitac.com.tw/google-cloud-platform/google-bigquery/load-json-data-to

● Streaming upload: https://developers.google.com/bigquery/streaming-data-into-bigquery

Quick Start● Features: https://cloud.google.com/products/bigquery#features● Case Studies: https://cloud.google.com/products/bigquery#case-

studies● Pricing: https://cloud.google.com/products/bigquery#pricing● Documentation: https://cloud.google.

com/products/bigquery#documentation

Google Computing Engine - Run virtual machines at Google Scale

Google Compute Engine

Features

● Scale, performance, and value● Flexibility and an Open Environment● Predictable Performance● Strong Security● Environmental Impact● Big Router● Global fiber

Google Style Management

● Fancy management console● Share permissions with Google Account● Tag for machine, ACL, routing● Software Defined Networking● Start Script● Mass technical documents share

Prepare

● google cloud project● google-cloud-sdk with gcutil● ssh tool

Reference:https://sites.google.com/a/mitac.com.tw/google-cloud-platform/google-compute-engine/gce---sdk-install-and-auth

● GCE architecture ● GCE web UI to GCE CLI tool

○ Create, Snapshot, Create from Disk or Snapshot● Network & FW

○ 3-tier network implements● Instance option - start script

○ Using start script build a auto scale service

Today’s Objective

Compute Engine Architecture

Network

Firewall

Instances

IP(Static, Dynamic)

L3 Load Balancing

Something about GCE

● Billing: 1 Minute Increments, Minimum 10 Minutes● Security:

○ ISO 27001:2005 Certification for GCE, GAE, and GCS ● Location:

○ Region○ Zone

About the Instances

● Persistent Disk● Network block storage● Max of 16 disks/instance● Created independently of instance● 1 Virtual CPU is a Hyperthread on Processor● Current processor is 2.6 GHz Intel Sandy Bridge Xeon● No GPU or SSD Options

About utility - web ui, gcutil, restful

Compute Engine Web UI

From Web UI to CLI to RESTful

gcutil - Get HELP

➔ gcutil --help➔ gcutil help listinstances

◆ ex: gcutil listinstances --columns=all --format=json➔ https://developers.google.com/compute/docs/gcutil/tips

RESTful APIs

https://developers.google.com/apis-explorer/#p/compute/v1/

Connect to GCE machine

➔ gcutil ssh [instance id]➔ ssh [username]@[instance-ip] -i [path-to-google-ssh-key]

Windows connect GCE

● Prepare ssh private key for project metadata [Ref]

[username]:ssh-rsa [private keys value]

Network & Firewall & Instance Scripts

Sample of create N-Tier

● Security purpose● Permission control● Management purpose● Tiers

○ admin: VPN, management purpose

○ frontend: web server, for public connect

○ db: storing data, sensitive area

http://gappsnews.blogspot.tw/search?q=n-tier

# service portgcutil addfirewall --allowed_tag_sources=frontend --network=my-network --allowed=tcp:80,tcp:443 myfw-service-port

# ap to dbgcutil addfirewall --allowed_tag_sources=frontend --target_tags=db --network=my-network --allowed=tcp:5984 myfw-couchdb-port

# admin zonegcutil addfirewall --allowed_ip_sources=0.0.0.0/8 --network=my-network --allowed=tcp:22 myfw-admin-ssh

gcutil addfirewall --allowed_tag_sources=admin --target_tags=frontend,db --network=my-network --allowed=tcp:22 myfw-manage-zone

Sample of create N-Tier - Network ACLs

Sample of create N-Tier - VPN & Web servergcutil --project="my-project" addinstance "my-gateway" \

--tags="admin" --zone="us-central1-b" --machine_type="g1-small" \

--network="my-network" --external_ip_address="ephemeral" \

--can_ip_forward="true" \

--image="https://www.googleapis.com/compute/v1/projects/.../global/images/..." \

--persistent_boot_disk="true"

gcutil --project="my-project" addinstance "my-web-01" \

--tags="frontend" --zone="us-central1-b" --machine_type="n1-standard-1" \

--network="my-network" --external_ip_address="ephemeral" \

--can_ip_forward="true" \

--image="https://www.googleapis.com/compute/v1/projects/.../global/images/..." \

--persistent_boot_disk="true"

Instance option - Start Script$ cat -> install-couchdb.sh << EOF

sudo apt-get update -y

sudo apt-get install gcc openssl couchdb -y

EOF

$ gcutil --service_version="v1" \

--project="my-project" addinstance "my-couchdb-01" \

--tags="db" --zone="us-central1-b" --machine_type="n1-highmem-2" \

--network="my-network" --external_ip_address="ephemeral" \

--can_ip_forward="true" \

--image="https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-6-v20131120" \

--persistent_boot_disk="true"

--metadata_from_file=startup-script:install-couchdb.sh

Share your project

● Is Owner: resource management, project permission● Can Edit: resource management● Can View: resource view

● Add persistent disk● Create image● Bring your own kernel (brief)

Advance operations

Add a Persistent Disk...

➔ gcutil adddisk --zone=us-central1-a testdisk➔ gcutil ssh [instance name]➔ sudo mkdir /mnt/pd0➔ sudo /usr/share/google/safe_format_and_mount \

-m "mkfs.ext4 -F" /dev/disk/by-id/[disk-id] /mnt/pd0

Create a Image...

➔ sudo gcimagebundle -d /dev/sda -o /tmp/ \ --log_file=/tmp/abc.log

➔ gsutil cp /tmp/308...439.image.tar.gz \ gs://arecord-customise-images

➔ gcutil addimage test-image \ gs://arecord-customise-images/308...439.image.tar.gz

Porting recommendation

● Install LAMPsudo yum -y install httpd php php-mysql mysql mysql-serversudo yum install php-mysql php-gd libjpeg* php-imap php-ldap php-odbc php-pear php-xml php-xmlrpc php-mbstring php-mcrypt php-bcmath php-mhash libmcrypt

● FW configure (GCE default enabled the iptables)sudo vi /etc/sysconfig/iptables⇒ Add your port… like 80, 443...

● SELinux setting (GCE default enable the SELinux)sudo vi /etc/sysconfig/selinux⇒ SELINUX=disabled

● Setup boot level servicessudo chkconfig --level 23456 mysqld onsudo chkconfig --level 23456 httpd on

Porting recommendation

● Mount persistence disk when boot$ sudo vi /etc/fstabUUID=a8cf...aaf98 / ext4 defaults,barrier=0 1 1

tmpfs /dev/shm tmpfs defaults 0 0devpts /dev/pts devpts gid=5,mode=620 0 0sysfs /sys sysfs defaults 0 0

proc /proc proc defaults 0 0

/dev/sdb /mnt/pd0 ext4 defaults 1 1

Bring Your Own Image

● Any common Linux distro● Must support some specific kernel settings (e.g.,

specific PCI and ISA bridge, vCPU settings, SCSI settings)

● Must have Python 2.6 or higher & sshd● Must contain some Google packages (startup script

support, google-daemon, gcimagebundle)● Should have other settings configured (e.g. DHCP,

SSH, firewall)

Quick Start● Features: https://cloud.google.com/products/compute-

engine#features● Case Studies: https://cloud.google.com/products/compute-

engine#case-studies● Pricing: https://cloud.google.com/products/compute-engine#pricing● Documentation: https://cloud.google.com/products/compute-

engine#documentation

More Requirement ?