Create a Big Data Cloud Instance - Tutorial

Before You Begin

Purpose

In this tutorial, you learn how to create a Big Data Cloud Service - Compute Edition (BDCS-CE) cluster for

various purposes. Prior to creating your BDCS-CE cluster, you will sign-up for the Free Trial and setup

your Cloud Storage service.

There are three sections in this tutorial:

Signing Up For Your Cloud Service Free Trial

Setting up Cloud Storage

Creating a Big Data Cloud Service

Time to Complete

60 minutes

Background

The Oracle Big Data Cloud Service – Compute Edition (BDCS-CE) enables you to rapidly, securely and

cost-effectively leverage the power of an elastic, integrated Big Data Infrastructure to unlock the value

in Big Data. In this tutorial, we will walk you through the steps to quickly configure and create a Big Data

Cloud Service instance. When done you will see how to view the configuration and layout of your

instance using the Oracle Big Data Console, the Ambari management interface and how to access your

instance using SSH to your servers using a command line terminal. The tutorial will also show how to

add additional nodes to a Hadoop cluster, monitor jobs, set access rules, modify configuration settings

and view activity on the Hadoop cluster.

Context

This tutorial is part of the New Data Lake series Oracle Big Data Journey. The sequence of modules in

the series is:

Module 1: New Data Lake Overview [Video] Module 2: Create Big Data Cloud Service - Compute Edition (BDCS-CE) Instance [Guided Tutorial]

o Module : Sign up for a Oracle Cloud Trial [Guided Tutorial] [Self Guided Tutorial] o Module : Create Object Storage Instance [Guided Tutorial] [Self Guided Tutorial]

Module 3: Working with BDCS-CE (Notebooks, Object Storage/HDFS, Spark and Spark SQL) [Guided Tutorial]

Module 4: Create Event Hub Cloud Service (OEHCS) Instance [Guided Tutorial] Module 5: Working with OEHCS and Spark Streaming [Guided Tutorial]

Signing Up For Your Cloud Service Free Trial

Let’s begin by setting up your Cloud Service Free Trial by performing the following steps:

1. Free trial – a credit of $300 2. Go-to cloud.oracle.com/tryit and click on Get Started for free 3. You will be prompted to either sign-in or create a new account. 4. If a new account has been created then the email needs to be verified 5. Once verified, click on Free trial again you will be prompted to enter your payment details 6. Select entity type as Individual and then add payment details 7. You will be charged $1 for credit card verification (which will be reimbursed). 8. Once payment details are updated, cloud services are provisioned and an email with service

activation details is sent to your registered email id.

Once you have completed the above steps, you are ready to launch the My Services Administration web

page in a browser.

Launch My Services

Click on the My Services Administration link provided in your activation email. It will be a URL similar to:

Example: https://myservices.us.oraclecloud.com/mycloud/a99999/faces/dashboard.jspx

Enter your username and temporary password (also in the email).

You will be prompted to change the password and set security questions/answers.

After successful identity operation, you will be taken to the Oracle Cloud My Services portal where all

available cloud services are listed. Take a moment to explore them.

Getting Started The first screen will give you a list of available cloud services you have access to. Available cloud

services may include Developer, Database, MySQL, GoldenGate, Compute, Java, Database Backup,

From: Oracle Cloud oraclecloudadmin_ww@oracle.com

Subject: Welcome to Oracle Cloud. Your Oracle Public Cloud Services Free Promotion is ready. Some services may require

additional configuration and provisioning steps.

Date: April 4, 2017 at 5:44 PM

To: TJLUCKENBACH@GMAIL.COM

Cloud

Hello Admin,

You have signed up for the Free Oracle Cloud Promotion. With this promotion, your start with a 300 USD Cloud Service cr edits in

your Oracle Cloud Services Account. This balance can be used towar ds activating and using any of the meter ed Oracle Cloud

Services in the following categories: PaaS, IaaS, Big Data and Middlewar e Cloud Services, which are available as Pay-as-You-Go

subscriptions.

You are the primary administrator of your Oracle Public Cloud Services. This includes the My Services Administration and the My

Account Administration. As of now , you are the only person who receives these cloud service notifications for your or ganization

until you add any other administrator fr om My Services.

We want to remind you that if you have set the intent to convert your account to a Pay-as-You-Go (paid) account as YES, your

credit card will be charged after the Free Oracle Cloud Promotion credits are used up, or for service usage incurr ed after the

promotion expires. You can change your intent at any time during the pr omotion period by accessing your Oracle Stor e profile.

If you have set your intent to convert your account to Pay-as-You-Go (paid) account as NO, the account will get suspended after

the Free Oracle Cloud Promotion credits are used up, or the promotion period expires. While your account is suspended, you will

have a grace period of 7 days to change your intent to convert to Pay-as-Y ou-Go (paid) account as YES before the account is

terminated. You can change your intent once during the grace period. Once you change your intent to YES during the grace period,

your credit card will be charged for service usage incurred thereafter.

You will receive notification alerts when: (1) Your balance reaches 75 USD, (2) You have 7 days remaining before your promotion

expires (3) Your Free Cloud Promotion balance is depleted, or your fr ee promotion expires after the 30-day duration. You can set up

additional alerts using My Services.

Getting Started

Your access details, including your user name, temporary passwor d and identity domain name are included below. As a service

administrator you can manage administrative functions r elated to your Cloud Services, including configuring and managing service

instances and performing monitoring and management tasks r elated to individual services.

Some services may require additional configuration and pr ovisioning steps before they're ready for production usage. See

Subscription details for additional information.

To start, use the Getting Started guided steps .

Contact Us

If you have questions or need help during the pr omotional period, you can chat with a live Cloud support associate .

If you need assistance with continuing or cancelling your Cloud services, please contact Sales.

Thank you.

Oracle Cloud Services Team

Access Details

Storage, Container, SOA, Event Hub and Big Data – Compute Edition. The focus of this tutorial is the Big

Data – Compute Edition Cloud service.

Note: Screens will use red boxes and arrows to highlight key areas for that step in the tutorial.

Setting up Cloud Storage At the top right of the screen there is a green button that says “Create Instance”. Click this green

button to start setting up your Cloud services.

Screen Displaying List of Cloud Services

A screen will pop-up providing a list of Cloud service instances that can be created. It may be necessary

to scroll down using the slider on the right edge of the screen. Click on the choice for the Storage option.

Figure 1 Screen Displaying List of Choices for Creating Cloud Storage

You will be taken to the Storage Cloud Service details page. Note the Identity Domain Name (such as

a457578) and write it down as you’ll need it later.

Click on Open Service Console button

Select the required replication region and click Set Policy

You will be taken to the Account Information page

Create Container Click Create Container

Give the container a name and click Create

Container “myContainer” is created. Remember the name of the container you created (ideally, write

it down somewhere).

You can now add objects (data) to it. We will re-visit this container again, later.

Go back to the myServices Dashboard. We will choose the Create Instance button again but this time we

will create a Big Data Cloud Service.

Creating the Big Data Cloud Service

Screen Displaying List of Cloud Services

Once the Big Data – Compute Editions is selected, the next screen will display a list of Big Data Cloud

services. The summary section at the top will display the list of services, OCPUs, Memory, Storage and

Public IPs associated with your Big Data Cloud services. This display screen shows an existing account

that has many Big Data instances.

Start by clicking on the blue button on the right that displays “Create Service”. Enter a Service Name,

Service Description and Metering Frequency for your Cloud service. Then click on the “Next” button on

the top right.

Screen to Create Service: Provide Basic Service Information

The Service Details screen contains the Cluster Configuration Information, Cloud Storage Credentials,

Security Credentials and Associations.

Screen to Create Service: Service Details Information

Cluster Configuration Information

Number of Nodes – Define the number of nodes for your configuration. Only add the number of

nodes you need. For this tutorial, we recommend you start with 1 node, more nodes can be

added later.

Compute Shape – From the drop down list, specify the compute shape (OC) for each node in

terms of OCPU and memory. For this tutorial, we recommend you start with 2 OCPU.

Usable HDFS Storage – Specify how much storage to allocate for HDFS. Once that is specified,

the display will show the Total Allocated Storage. For this tutorial, we recommend you start

with 50GB.

Queue Profile – This will determine if Queue Preemption is enabled or not for YARN. YARN

handles resource management for the Hadoop cluster. If Preemption is enabled, under-served

YARN queues can claim their allocated resources immediately if there is contention for

resources.

Historically the default setting for HDFS block replicas has been set to 3 copies. BDCS-CE uses

network-attached block storage underneath HDFS. The block storage itself is protected by RAID

(multiple copies). So, BDCS-CE's HDFS layer uses a replication setting of only 2 (where each of the 2

copies is protected by multiple copies thanks to the underlying RAID).

Screen Section for Cluster Configuration

Credentials

SSH Public Key – Define a value for a VM Public Key, use a file with a VM Public Key or create a

new key. The easiest choice if new to this environment may be to create a new key. Choose to

Create a New Key and hit the Enter button. Once you hit Enter, a File Folder Window will pop

up to allow you to control where on your local computer you wish to store your SSH Key file (ex:

sshkeybundle.zip). Make sure and write down the location of this SSH key file. The SSH Public

Key field will then get filled in automatically. Also store safely the administration user id and

password to be entered next.

Administrative User – Define the user id for the administration user for your instance.

Password – Enter a password for the administration user.

Confirm Password – Verify the password for the administration user.

Screen for SSH Public Key for VM Access (popup)

Cloud Storage Credentials

Here you can use Cloud Storage if the storage had been previously set up.

Under Cloud Storage determine the Cloud Storage Container to use. Then under the Rest Endpoint field,

perform a copy/paste back into the Cloud Storage Credentials page. Then enter the Cloud Storage

Container, Username and Password given to you in the email you received regarding your Cluster

Instance.

Cloud Storage Container – This will be of the format: Storage-IDENTITYDOMAIN/CONTAINER,

where IDENTITYDOMAIN is something like “a999999” and CONTAINER is something like

“myContainer”. For example: Storage-a999999/myContainer

Username – Your cloud username

Password – Your cloud password

Create Cloud Storage Container checkbox – Check if you do not have a container and need to create

one. For this tutorial, you should not need to create a container as you created one called

“myContainer” earlier in this document.

Screen Section for Cloud Storage Credentials

Service Details: Oracle Storage Cloud Service (Finding your Cloud Storage Container)

Associations

Here you can specify addition Cloud services that can be used with your Big Data Cloud Service such as

an Oracle Database Cloud Service, MySQL Cloud Service and/or Event Hub Cloud Service. This self-

guided tutorial does not set up any associations for this Big Data Instance. An Event Hub Cloud Service is

set up in a different module later as part of the New Data Lake journey.

Screen Section for Associations information

After filling out the Cluster Configuration Information, Cloud Storage Credentials, Security Credentials

and Associations information, then click the Next button located towards the top right of the screen.

After selecting Next, you the Confirmation screen will appear next. Look at the Cluster Configuration

Information, Cloud Storage Credentials, Security Credentials and Associations information to verify it is

what you wish to use for creating your instance. If something needs to be corrected, click on the

Previous button on the top left of your screen. If everything looks ok, then select the Create button

towards the top right of your screen. Note, it will take a few minutes to create the Big Data Cloud

instance.

Screen for Confirmation of Configuration

Once your Big Data Cloud instance is created you will be taken back to the BDCS-CE Services page where

you will see your instance. While your instance is being created the status will be “Creating service”.

Once the new Cloud Service instance changes its status to Ready, you can then click on the hyperlink

with your Service name and you can look at the details of your new Service. Sections include:

Overview – displays the number of nodes

Administration – displays if there are any patches available.

Services Overview – displays summary information of the new Big Data Cloud Service. This

includes the Ambari Server Host whose IP address you can use to access Ambari from a URL in a

browser. As well as highlighting the Administrative user you created as well as the Cloud

Storage Container and the Spark Thrift Server (part of the default configuration). Ambari is a

Hadoop management web UI that can accessed through your Ambari Host Server IP address and

port 8080 (ex: http://xxx.xxx.xxx.xxx:8080), then use your admin username and password to log

in.

Resources – displays information on the resources associated with your Service. As you scale

out and add more nodes, the new nodes as well as their Public IP address, OCPUs, Memory and

Storage will be displayed.

Associations – displays information on any additional resources associated with your Service.

Screen showing Services

This screen contains the Services, Activity and SSH Access tabs. While waiting we can click on the SSH

Access tab to look at our SSH Public Key information.

When the Big Data instance is created, the status will change and you will see the following change in

this section. There will be a Created date and time, storage will be available and the Version will display

without the Status of Creating Service.

After clicking on the SSH Access tab we can see our Service name for our SSH Public Key. Notice the Add

New Key button is greyed out. Once the Big Data Cloud instance is created, this button will no longer be

greyed out. It is possible to click on the Add New Key button and use a separate Public Key file. Note

the screen below shows multiple service names. If this is your first instance you will not see a larger list.

Screen showing the SSH Access Information

If someone decides to use a different SSH Public key file, they can click on the Add New Key button and

select a new SSH Public key from a different file. Click on the Browse button to select a new Public Key

field. Then click on the Add New Key button on button right to use a different Public Key.

Screen Displaying the Add New Key screen (popup)

You can look at the details of our Big Data Cloud Service instance.

In the Access Rules screen you can review the current access rules or create new rules.

Screen Access Rules

Information showing the Status is ready, the Ambari Server Host Public IP address, configuration

information, etc. is displayed.

Screen showing information for instance created.

At the top of the screen, next to the Big Data Cloud Service instance name (George-BDCSCE), there is a

grey icon that looks like a stack of pancakes that will display a list of options when clicked. Shown

below (Screen showing information for instance created.)

These options will take to the Big Data Cluster Console, allow you to Start, Stop and Restart your

instance, Scale Out the cluster with additional nodes, define Access Rules, configure SSH Access and

View activities. To the right of the grey icon is a set of larger blue icons that allow you to Start, Stop,

Restart the cluster, Scale out and look at the Big Data Cluster Console. So you have a choice of clicking

on the grey icon to get a larger list of options or click on the blue icons to get to an operation with one

click.

Screen showing additional options after clicking grey icon

Clicking on the grey icon mentioned above brings up a list of options, one of which is Access Rules. This

shows the list of access rules currently defined. As mentioned previously, new rules can be created by

clicking on the Create Rule button. Note both SSH and Ambari have a red X to the left of them. Go to

the Actions column on the far left to enable them. This is shown below.

Screen showing additional options after clicking grey icon

Along with the Big Data Console and the Ambari Management UI, a tool like Putty can be used to SSH

into one of your servers using their Public IP address. Just remember in the tools Authorization section

to add the Public Key file for access. Then use the administrator username and password to log in and

get terminal access to any of your servers. It may be necessary to use a tool like Putty to convert the

Public Key file to a format the tool understands. For example, the Putty Key generator can be used to

convert the Public Key file to a format Putty can work with. In Putty go to the Conversions Menu in the

Putty Key generator tool and import the Public Key file and perform the conversion.

A terminal tool like Putty can then be used to perform an SSH into one of the servers in your cluster.

Screen setting the Public Key in Putty.

You can also click on the Icons on towards the top-right to Start, Stop, Restart, Add a Node and Monitor.

Big Data Console – can used to specify the admin username/password to log in to the admin Console.

Once you log in you can see a Summary, HDFS Capacity, CPU Usage, Memory Usage as well as look at

status of Jobs, Datastores, look at Settings and work with Notebooks. In Settings you can look at

settings for Queues, Credentials and Notebooks.

Screen Displaying Activity for the Big Data Instance

Want to Learn More? Need some recommended URLs to point readers to? (help)

https://docs.oracle.com/cloud/latest/big-data-compute-cloud/CSSPC/toc.htm URL2

…