cloud computing: what a project manager needs to know
DESCRIPTION
Cloud Computing: What a Project Manager Needs to Know. Dr. Patrick D. Allen, PMP [email protected]. Purpose. Provide Project Managers with the very basics of the two primary types of Clouds and Cloud Computing, and the questions they should ask when Clouds and their project intersect. - PowerPoint PPT PresentationTRANSCRIPT
Purpose
Provide Project Managers with the very basics of the two primary types of Clouds and Cloud Computing, and the questions they should ask when Clouds and their project intersect
2
Overview
“Computing as a Service” Clouds Questions PMs should ask
“Data-Focused” Clouds Relational Databases vs Clouds Map-Reduce and Accumulo examples Questions PMs should ask
General Cloud questions PMs should ask The importance of risk assessments
3
What’s a Cloud? Two primary definitions of Clouds presented today:
1. Compute-power as a Service (Utility Cloud; VMs) Infrastructure as a Service or Platforms as a Service or Software as a Service
2. A Data-focused Cloud that also runs on VMs E.g. Hadoop Data File System and data processing
A third emerging type is a “Data Storage” Cloud PMs need to make sure everyone understands which type
is being discussed If you think you’re discussing a different one, confusion
results and expectations will not be met
4
First Type: Computing as a Service Instead of using your own computers, you use a Third-
Party’s computers at another location (e.g., AWS’s EC2) Usually all same hardware with a variety of Virtual
Machine (VM) configurations to meet customer needs When hardware dies, it is seamlessly replaced All hardware and infrastructure and physical security
headaches are the responsibility of the Third Party You’re responsible for secure comms to and from the
data stores and the security on the machines you use You only pay for what you use (memory, computing
power or number of virtual machines used) Great for surge-type activities, such as the census
that’s run every ten years, or new venture start-ups Virtual private clouds are available for better security
5
First Type: Questions PMs Should Ask –1
What’s the cost per data stored (Cents per Gigabyte)? What’s the cost for number of VM’s used? How secure or private is my data when I store it on a
third-party platform? What security or privacy guarantees are provided? Will the PII be adequately protected? Can I test Cloud security before I put real data there? Would a Cloud be useful for my Continuity of
Operations (COOP) plans? It depends. Do your employees already regularly
perform remote operations like teleworking? Do you have a re-routing plan to get them to the Cloud?
Am I starting a new business with limited investment? 6
First Type: Questions PMs Should Ask – 2 Can you store classified data on a cloud?
If a properly secured government-accredited private cloud, Maybe If you are planning to use a Third-Party service, Maybe
As a minimum, use a virtual private cloud (e.g., AWS VPC) And located entirely in the U.S. (not distributed world wide) Probably need to limit access to selected personnel at the service
provider site (like no foreign access in US Gov Cloud) US-Gov-only Cloud important for data under export control Need your security department’s approval, which includes your
plan and vetting the provider Probably need to do penetration testing before use, like “side
channel attack” prevention Not sure if this is yet being used for more than unclassified but
sensitive data For either case, always get a cyber security expert to prepare a risk
assessment, and for classified data, a proper accreditation
7
2nd Type: Data-Focused Cloud–Definitions Huge Data: Petabytes or larger amounts of data HDFS is Hadoop Data File System (more on this later) Relational Database: Think rows and columns, densely
populated (like a spreadsheet) Structured non-relational databases: Cloud-based structured
data technologies like Accumulo and HBase running on HDFS Can be densely or sparsely populated Tend to use flexible labels of length three to six (more later) Many different types of data that may have some overlapping
elements, but not the same across all types of data If put into rows and columns it would be a huge table only
sparsely populated
8
Relational Database Example
9
Name Address Age HeightJohn Smith
Jane Doe
Fred Flintstone
Tony D. Tiger
Elmer Fudd
Peter Parker
Bruce Wayne
Roger Rabbit
Peter Rabbit
White Rabbit
Washington DC
Baltimore
Rockville
Battle Creek
DeForest
New York
Gotham
Fantasyland
Rural Address
Wonderland
35
29
55
67
60
28
36
41
118
135
5’10”
5’8”
4’10”
6’2”
4’6”
5’5”
6’1”
4’0”
1’1”
1’11”
Find the Names of those of Age >25 but <60, and > 5’ tall
Sparse Data Example
10
John Smith
Jane Doe
Peter Parker
Bruce Wayne
Washington DC
Baltimore
New York
Gotham
Age 35
Age 29
Age 28
36
5’10”
5’8”
5’5”
6’1”
Medical Records Drivers Licenses Facebook Dating Service
John Smith
Peter Parker
Bruce Wayne
Find the Names of those of Age >25 but <60, and > 5’ tall from multiple data sets
Accumulo Data Example
11
ID Col. Family Time Security ValueCol. Qualifier001001001001001001
Personal Name 31 Apr ‘12 PII John SmithPersonal Age 31 Apr ‘12 PII 35Personal Height 31 Apr ‘12 PII 5’ 10”Address City 31 Apr ‘12 PII Wash DCAddress Street 31 Apr ‘12 PII K StreetAddress Number 31 Apr ‘12 PII 810
002002002002002002
Personal Name PII Peter ParkerPersonal Age 31 Apr ‘12 PII 28Personal Height 31 Apr ‘12 PII 5’ 5”Address City 31 Apr ‘12 PII New YorkAddress Street 31 Apr ‘12 PIIAddress Number 31 Apr ‘12 PII
72nd Street145
31 Apr ‘12
Find the Names of those of Age >25 but <60, and > 5’ tall
2nd Type: Data-Focused Cloud Also runs on a VM farm, but uses a “Hadoop” or “Sector”
file management system (Hadoop is most widely used) What does a Hadoop Data File System (HDFS) do for you?
Let’s you store huge amounts of non-relational data Automatically parallelizes the computations Automatically sorts results of “map” step Handles all of the overhead associated with storing,
locating and processing your data Allows for Map-Reduce programs and Direct Access
Table-based searches using Hadoop to be run Can find relationships not easily visible in unstructured
data and/or large amounts of data
12
2nd Type: Map-Reduce Program Example Find the number people per household in census data
13
DistributedDatabases of Household (HH)Census Data
Count members of HH
Hadoop Auto Sorts
Map Reduce
Add # HHw/ N members,N = 1 to 25
1, 3.5 M
2, 9.6 M
3, 6.8 M
4, 5.3 M
Key = HH Size, Value = #
HH001, 3
HH002, 6
HH003, 4
HH004, 3
HH001, 3
HH002, 6
HH003, 4
HH004, 3
Key = #, Value = Total
2nd Type: Map-Reduce Pros and Cons Map-Reduce programs are good for:
When you have huge data sets If your data can't be managed in a relational database When you are not sure what types of queries you will want to run If you want to summarize the results of independent processes
that can be applied to data in parallel
Map-Reduce programs are not good for: If you can answer your questions with an existing relational
database in a reasonable amount of time, why bother with the overhead of a cloud?
If your data can fit within a relational database, AND If the queries you plan to run are fairly well-defined THEN You probably don’t need the overhead of a Cloud
14
2nd Type: Questions PMs Should Ask – 1 Do I even need to use a Cloud?
If you have well-structured reasonable amounts of data, stick with a relational database UNLESS you just want the compute power on demand (1st Type of Cloud presented)
If it is required by external authorities (like a customer), yes Do I have a lot of "surge" events, where you only need to store and
process large amounts of data periodically Then using a cloud makes sense
Do I need to know how to write a Map-Reduce program or an Accumulo Table to use a Cloud? No, can use pre-defined programs, OR you need someone who
knows how write new ones for you Do I need to know how to design a Map-Reduce program?
No, but it helps so you can ask for realistic output from the Cloud and really leverage the Cloud to solve your data problems
15
2nd Type: Questions PMs Should Ask – 2 Do I have access to an existing Cloud I could use?
If it meets your requirements, third-party Clouds work Make sure of the “fine print” on the guarantees, and whether the
recourse of the guarantee is sufficient to match the cost of the failure to guarantee
Have a security expert do a risk assessment before committing Do I need to build my own instead?
If you have security, privacy or proprietary needs not met by an existing Cloud, might want to build your own
Consider the ongoing maintenance costs (may be primary rationale for moving to a Cloud)
More automation reduces Cloud maintenance costs
16
General Cloud Questions for PMs Where is the Cloud located? Can it be restricted to U.S.? Who gets access to it? How are the communications to/from the cloud secured? How does it ingest its data? How does it store its data? How do they secure your data at rest? How does it delete its data? Can you test that it’s gone? Does it keep your data separate from other people's data?
Do you need/want a virtual private cloud instead? How often is the hardware upgraded? How many versions of VMs can you choose from? Has a security expert performed a risk assessment?
17
Summary Observations Cloud computing is here to stay Many more projects in the future will encounter Clouds in
some way that will impact the project Need to be aware of the strengths and limitations of
Clouds and whether they are appropriate for your project You may not have a choice whether or not to use a Cloud
This briefing listed some of the basic questions you should ask as appropriate to your project
Hopefully some of the mystery (and hype) of the Cloud has been dispelled by this talk
It is useful to be able to design a Map-Reduce program so your expectations of the output are realistic
Always do a cyber risk assessment on a Cloud you plan to use
18
Contact InfoDr. Patrick D. AllenJohns Hopkins University Applied Physics Lab11100 Johns Hopkins RoadMS 21-N246Laurel, MD 20723-6099443-778-9915 v443-778-3838 [email protected]
19
New in This Edition (14 Aug 2012)
FedRAMP is a new standardized approach to security assessment, authorization and security monitoring for cloud-based products and services
FedRAMP is mandatory for federal agency cloud deployments and service models at the low and moderate risk impact levels
Ref: The Business Monthly, Aug 2012 by Gloria Larkin “Cybersecurity and FedRAMP: A Mandatory Combination
20
Back-up: Terminology Relationship
21
Google File System (GFS)
Hadoop Data File System (HDFS)
Hadoop(Map Reduce)
Map Reduce
Big TableHDFSAccumulo
APACHE GOOGLE
StructuredData
Map Reduce Environment
File System
Back-up: Sample Map Reduce Program
22
Map algorithm
Map (key: sourceURL, value: text) {for each (targetURL in text)EmitIntermediate (targetURL, sourceURL);
}
Reduce Algorithm
Reduce (key: targetURL, value: sourceURL) {sourceList[] = null;for each (u in sourceURL)add sourceList[sourceURL];Emit (targetURL, sourceList[]);
}
Back-up: Map Reduce Example 2
23
Find targets for source 1
Find targets for source 2
Find targets for source 10^9
targetURL a – URL1
targetURL b – URL1
targetURL a – URL2
targetURL c – URL2
targetURL b – URL10^9
targetURL c – URL10^9
targetURL d – URL10^9
targetURL a – URL1
targetURL a – URL2
targetURL b – URL1
targetURL b – URL10^9
targetURL c – URL2
targetURL c – URL10^9
targetURL d – URL10^9
Create list for targetURL a
Create list for targetURL b
Create list for targetURL c
Create list for targetURL d
sortedtargetURL – sourceURL list
Doc 1
Doc 2
Doc 10^9
For each URL, find all the pages that point to it.