cloud computing lecture 11 private and hybrid clouds · 2020. 4. 20. · •abiquo hybrid cloud...
TRANSCRIPT
-
Cloud Computing – Lecture 11
Private and Hybrid Clouds21 April 2020
Shivananda Poojara
Pelle Jakovits
Satish Srirama
-
Outline
• Private clouds – Definition and architecture– Necessity and design considerations– Tools: Openstack, Cloudstack,OpenNebula– advantages and disadvantages
• Hybrid Clouds – Definition and architecture – Necessity and design considerations.– Hybrid cloud tools: ManageIQ, scalr– Advantages and disadvantages
• Hybrid cloud case studies using Aneka
Shivananda Poojara 2/34
-
Private Cloud
– Private clouds are on demand resources managed under single organization hosted in a dedicated physical infrastructure.
– Requires CAPEX and OPEX
– Size: depends on the need.
– Managed inside the organizational firewall
– More control and greater security
– Requires skill set in the organization
– Managed by organization, third party deployed on premise or third party locations.
Shivananda Poojara 3/34
-
Essential characteristics of private clouds
• On demand self-service– Automated provisioning
• Broad network access:– Access through multiple devices
• Rapid elasticity :– Resources scaled quickly and transparent manner.
• Measured service– Monitor, control and reporting
• Location-transparent resource pooling for multiple tenants– Sense of location independence
Source: https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-146.pdf
Shivananda Poojara 4/34
https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-146.pdf
-
Private cloud architecture
DataSynapse, Zimory Pools, Elastra CloudServer, Aneka, ….
Platform as a Service Solutions
Paa
S
IaaS
IaaS
(M)
Datacenters, Clusters, Desktop Grids
Physical Infrastructure
KVM, Xen, VMWare, …
Virtual Machine Technology
Eucalyptus, OpenNebula, VMWarevCloud, OpenPEX, InterGrid, …
Infrastructure Management Software
Source:buyya.com/masteringcloudcomputing
Shivananda Poojara 5/34
-
Necessity of deploying private clouds
• Workload and Infrastructure integration
• Security / Regulation/Compliance
• Latency
• User experience
• Cost
Shivananda Poojara 6/34
-
Private cloud design considerations
• Capacity planning – Hardware Considerations– Compute, Storage and network
• Software considerations – COS- Cloud Operating system , Hypervisor.
• Physical location of data canter– On premise, third party location or near to public
clouds
• Implementation process – Design, automate, monitor and test
• Cloud design considerations- HA
Shivananda Poojara 7/34
-
Private cloud tools-COS
• A cloud operating system(COS) manages the operation, execution and processes of virtual machines, virtual servers and virtual infrastructure, as well as the back-end hardware and software resources.
• Tools-– Openstack
– Cloudstack
– Eucalyptus
– OpenNebula
Shivananda Poojara 8/34
-
OpenStack Private Cloud Platform
• Opensource• Written in : Python• Collection of open-source software tools, components and services,
called OpenStack Services• Service-Service Interaction through public APIs*• provides an Infrastructure-as-a-Service (IaaS) solution• Access:
– via Web-based user interface implemented by Horizon Dashboard– via Command-line client– REST API
• IBM Cloud Orchestrator works with the open source OpenStack software
Shivananda Poojara 9
* Except where privileged administrator commands are necessary
ppt 9-25 prepared by Chinmaya Dehury
-
CloudStack Private Cloud Platform [6]
• Developed by: Apache software foundation• Open-source cloud computing software
– creating, managing, and deploying infrastructure cloud services
• Written in : Java, Python• Access:
– via Web-based user interface – via Command-line client– REST API
• Support communicating with AWS EC2 and S3, for hybrid deployment.
Shivananda Poojara 10
-
CloudStack Private Cloud Platform [6]
• Supported Hypervisor:
– KVM-RHEL, LXC, XenServer, HyperV, etc.
• Users:
– Apple, China Telecom, British Telecom, Dell, Disney etc.
Shivananda Poojara 11
-
OpenNebula
• Open source, enterprise-ready solution to build your Elastic Private Cloud
• Written In: C++, Ruby, Shell Script, etc…
• Two primary uses:
– Data center virtualization solutions
– Cloud infrastructure solutions
• Cloud Management on VMware vCenter
Shivananda Poojara 12
https://github.com/OpenNebula/one
-
OpenNebula: Architectural Overview
Shivananda Poojara 13
Basic components of an OpenNebula system :• Front-end that executes the OpenNebula services.• Hypervisor-enabled hosts that provide the resources needed by
the VMs.• Datastores that hold the base images of the VMs.• Physical networks provide communication service.
-
OpenNebula: Architectural Overview
Shivananda Poojara 14
Memory 8 GB
CPU1 CPU (4 cores)
Disk Size 100 GB
Network 2 NICs
• Single OpenNebula instance can manage a maximum of 2,500 servers and 10,000 VMs.
• OpenNebula front-end specification
-
OpenNebula: Architectural Overview
Shivananda Poojara 15
Storage:• Image datastore: to stores all the images• System datastore: to store the disks of VM’s• File Datastore: a special datastore used to store plain files, not
disk images. • Default database: sqlite
-
OpenNebula: Architectural Overview
Shivananda Poojara 16
Networking• At least two different physical networks are needed
• Service Network: used by front-end daemons• Instance Network: offers network connectivity to the
VMs across the different hosts.
-
Private cloud- Advantages
• Cost – TCO will be less for prolonged workloads
• Efficiency and control– Managed and tuned in on premise and have higher control
• Customize – Customize based on your business requirements and full
flexibility
• Security and privacy– Within company’s firewall and greater security
• Compliance– National and international laws and policies . Ex- data protection
• Location independence
Shivananda Poojara 17/34
-
Private cloud- Disadvantages
• Cloud bursting, scalability issues
– Ability to handle unpredicted workloads or new workloads
• Expertise
– Skillset and other logistics
• Scaling of hardware
• Initial investments and future pricing
– Higher CAPEX
Shivananda Poojara 18/34
-
Hybrid Clouds
Shivananda Poojara 19/34
-
Hybrid clouds
• Hybrid clouds is combination of on premise infrastructure with off premise resource or public clouds such as AWS, IBM,GCP or Microsoft Azure.
• Cloud bursting, Shrink by provisioning external resources when required and unleased when not needed.
• Perspective and people:
• Flexibility• Speed• Capacity• Consistent Agility
• Cost• SLA• Easy invoicing• Software leciening
• Visibility• Control• Manage• Access• Monitor
Hybrid Cloud
Solution designer team Infra TeamBusiness Team
Shivananda Poojara 20/34
-
Necessity of hybrid cloud
• Use Case 1: Untested workloads with uncertain demand.
• Use case 2: Cloud bursting
• Use 3: Disaster recovery and high availability
Private CloudPublic Cloud
Database servers
AS1 AS2
AS: Application Server
Load balancer
End point
AS3 AS4
Shivananda Poojara 21/34
-
Hybrid cloud architecture
Desktop Grids / NOWs
Private Cloud
DynamicProvisioning
Datacenter Cluster Desktop Grids / NOWs
Public Cloud
Enterprise dad center
Applications
PaaS
IaaS Middleware
Ref: Rajkumar et.el Mastering Cloud Coputing
Third party services
Shivananda Poojara 22/34
-
Hybrid Cloud: VmWare and Microsoft Azure Integration
ESXi Hosts
Storage Network
VM VM VM VM
Vmware vCenter
On Premise Cloud
ESXi Hosts
Storage Network
VM VM VM VM
Vmware vCenter
Azure bare metal servers
Azure Portal
Azure Management
Azure services
Azure Integration service
Source: https://www.harbourit.com.au/blog/everything-you-need-to-know-about-hybrid-cloud/
Shivananda Poojara 23/34
https://www.harbourit.com.au/blog/everything-you-need-to-know-about-hybrid-cloud/
-
Key Strategies for Hybrid Cloud Success
• The cloud operating system that gives you the ability to monitor, manage, and orchestrate across all environments using a single set of tools.
• Choosing a framework a cloud Operating systems like opens tack, cloud stack that will allow to manage workloads on-premises and in the cloud.
• Modernize your on-premises environments in accordance with that frame‐ work.
• Choose only public clouds and CSPs compatible with that framework.
Sourcehttps://cdw-prod.adobecqms.net/content/dam/cdw/on-domain-cdw/brands/nutanix/o-reilly-ebook-designing-and-building-a-hybrid-cloud.pdf
Shivananda Poojara 24/34
https://cdw-prod.adobecqms.net/content/dam/cdw/on-domain-cdw/brands/nutanix/o-reilly-ebook-designing-and-building-a-hybrid-cloud.pdfhttps://cdw-prod.adobecqms.net/content/dam/cdw/on-domain-cdw/brands/nutanix/o-reilly-ebook-designing-and-building-a-hybrid-cloud.pdf
-
Capabilities required -hybrid cloud
• Integration: Integration of data and processes needs to be seamless across the deployments,
• Data localization: Security and compliance• Operational visibility and management: Unified
management • Security services: Policies and business rules are needed to
understand and monitor information about workloads • DevOps: Single unified way to manage and synchronize
applications• Integration services: Approaches that allows application
code to be easily moved from one environment to another.
Source: https://www.ibm.com/downloads/cas/WEWZGM5J
Shivananda Poojara 25/34
https://www.ibm.com/downloads/cas/WEWZGM5J
-
Possible Workloads in hybrid cloud
• Workload- is an independent service or collection of code that can be executed.
• Workloads need to be executed based on: Criticality, Level of security, interaction with other environments.
• Types of workloads– Batch - customer bills and account statements
– Analytic – high volume, complexity, and importance
– Transactional- automation of business processes
– Database- high amount of Input/Out (I/O) cycles
Source: https://www.ibm.com/downloads/cas/WEWZGM5J
Shivananda Poojara 26/34
https://www.ibm.com/downloads/cas/WEWZGM5J
-
Assessing the needs of hybrid cloud
• Assessing the current state of operations– Identify the current IT infra locations.– Identify the status of infra like utilization, cost, physical space.– Identify the current cloud provider impact on your business.– Identify workload characteristics of your business.
• Assessing the future workloads:– Identify new applications, new products planning in a year.– Planning any updates for business applications.– Prioritize the workloads and assess type of application, resource
requirements, data locations.
• Create decision matrix – Decision matrix can help to determine where to locate each
workload going forward and how to assess anticipated costs.
Shivananda Poojara 27/34
-
Use cases
• Health care sector : Security and compliance in storing and movement of health data.
• Finance sector: Flexibility, power, scale, and seamless connectivity.
• Government: Use cloud email and collaboration tools and a mixture of public and private cloud infrastructure.
• Retail: Always in (99.999%) uptime support in ecommerce operations.
• ecommerce: Enormous web traffic and scalability.
Source: https://www.harbourit.com.au/blog/everything-you-need-to-know-about-hybrid-cloud/
Shivananda Poojara 28/34
https://www.harbourit.com.au/blog/everything-you-need-to-know-about-hybrid-cloud/
-
Hybrid cloud offerings and tools
• Microsoft Azure – Azure public cloud and on-premise Azure
Stack
• Amazon Web Services– AWS Outposts
• VMWare
• Google Cloud Platform
• RackConnect
• VMware-based private cloud
• Dell EMC
• IBM Cloud
• CloudBolt
• Cisco Cloud Center
• DivvyCloud
• RedHat Hybrid Cloud
• Abiquo Hybrid Cloud
• Cloudability
• Scalr
Shivananda Poojara 29/34
-
ManageIQ
• Its an open source project started in 2014 and managed by Redhat Inc.
• Other companies such as Xlab, Google,Lenovoare also part of the community.
• 43965+ commits, 256+ developers.
• Discover, Optimize, and Control hybrid cloud resources
• Manage containers, virtual machines, networks, and storage from a single platform
Shivananda Poojara 30/34
-
ManageIQ
It has the following feature sets:
• Insight: Discovery, Monitoring, Utilization, Performance, Reporting, Analytics, Chargeback, and Trending.
• Control: Security, Compliance, Alerting, Policy-Based Resource and Configuration Management.
• Automate: IT Process, Task and Event, Provisioning, Workload Management and Orchestration.
• Integrate: Systems Management, Tools and Processes, Event Consoles, CMDB, RBA, and Web Services.
Shivananda Poojara 31/34
-
High level architecture
Shivananda Poojara 32/34
-
Features: Self service delivery
• Service delivery catalog ,Multiple service at one time ,Approval routing
Shivananda Poojara 33/34
-
Features: Automated provisioning
• Automatic configuration and deploy
• Ansible playbooks or native coding
• Integration to external IT systems
Shivananda Poojara 34/34
-
Features: Transparent Operational Watch
• Continues discovery
• Relationship view
• Monitoring and tracking
• In depth analysis
• Analytics
Shivananda Poojara 35/34
-
Life cycle management
• Ongoing tracking, Scheduled retirement and archiving, root cause analysis
Shivananda Poojara 36/34
-
Other features
• Root cause analysis
• Performance and capacity management
• Policy enforcement
• Quotas and chargeback
• Virtualization management
• Container management
Shivananda Poojara 37/34
-
Scalr
• Its also hybrid cloud management software for enterprises
• It has highly efficient role and policy based control for security and cost management.
• It also have self service portal, power console, Service catalog, policy engine, multiple service provider support .
• Varity of deployment support such as HA,scaling
Shivananda Poojara 38/34
-
Advantages of hybrid clouds
• Flexibility : The ability to distribute workloads across public and private environments based on security, efficiency, and cost.
• Scalability: Dynamically the resources based on the demand spike and release when not required.
• Reliability: Less possible downtime even when demand spikes.
• Security: Specific workload distribution of sensitive data on private clouds and non sensitive data on public clouds.
• Affordability: Need not to purchase and manage new resources to handle short spike in demand.
Shivananda Poojara 39/34
-
Advantages of hybrid clouds
• Continuity - Business continuity is not interrupted during a failure or disaster, as data is still accessible with little or no downtime.
• Opportunity: New prototypes or new applications can be tested and deployed rapidly.
• Accessibility: Any time, any where
Shivananda Poojara 40/34
-
Disadvantages of hybrid cloud
• Possibility of vendor lock in
• Service integration issues
• Interoperability issues
• Transparency
• Security and privacy issues
• Data locality issues
Shivananda Poojara 41/34
-
Benefits of hybrid clouds
• Flexibility and agility
• Elasticity
• Self-service
• Faster delivery of new products and services
• Cost control
• Avoidance of lock-in
• Access to the latest technology
Shivananda Poojara 42/34
-
Hybrid cloud- Case study with Aneka
Shivananda Poojara 43/34
-
Hybrid cloud research challenges
• Resource management and scheduling in hybrid clouds
– Metrics: QoS, SLA, Failure, profit
• Interoperability challenges
• Data replication and sharing
• Security and privacy
• Disaster recovery and high availability
Shivananda Poojara 44/34
-
Aneka – Tool for Cloud Computing research
• Aneka is a .net based cloud application development platform developed by Manjrasoft Pty Ltd Melbourne Australia(Spin-off company of University of Melbourne)
• It supports deployment of three types of application
– High Throughput Computing(Task programming API)
– High Performance Computing (Thread Programming API)
– Data intensive computing(Map Reduce Programming API)
• Aneka can de deployed in heterogeneous infrastructure
– Private Clouds, Hybrid Cloud, Local desktop clutters, Virtualization clusters
• Develop custom resource scheduling algorithms
• It has ability to connect to multicloud environments
– AWS, Azure, openstack
Shivananda Poojara 45/34
-
High level view of Aneka
Shivananda Poojara 46/34
-
Aneka API’s and its working
• Aneka Thread API
• Aneka Task API
• Aneka Map-Reduce API
Shivananda Poojara 47/34
-
Aneka job submission
public DumbTask: ITask
{
…
public void Execute()
{
……
}
}
for(int i=0; i
-
Aneka support for hybrid cloud
• XenServer Pool
– Provisioning over private Cloud managed by Xen Server
• VMWare Pool
– Provisioning over private Cloud managed by VMWare
• Amazon EC2 Pool
– Provisioning over public Cloud provider: Amazon EC2
Shivananda Poojara
Executors/Schedulers
Executor
Client Libraries
Executors
Scheduler
Public Cloud
publicly available resources(physical and virtual)
Scheduler
internet
Private Cloud
private enterprise network
VPN(virtual resources)
Executor
49/34
-
Hybrid Cloud Case study: Aneka’s dynamic provisioning with Amazon EC2
Source:buyya.com
Use Case : Scheduling problem based on cost and time
Shivananda Poojara 50/34
-
Hybrid Cloud Case study: Aneka’s dynamic provisioning with Amazon EC2
Scheduling Service
Master Node
Tasks +
QoSProvisioning
Service
Public Clouds
Private Clouds
Static Resources
Worker Nodes
Dynamic Resources
Deadline+
Budget
Deadline Execution time Cost
80 min 70 min U$ 0,06
70 min 60 min U$ 0,12
60 min 55 min U$ 0,15
50 min 50 min U$ 0,15
40 min 35 min U$ 0,30
AWS Spot Instances,Small
Source:buyya.com
Shivananda Poojara 51/34
-
Practical session
• We will work Aneka programming in the lab
– Aneka deployed in open stack cluster consisting of 3 VM’s.
– You will work Aneka Task and Thread API.
– Focus of lab is to deploy application on anekacluster such as image processing and some custom thread programming application.
– Bit knowledge on .net but not compulsory.
Shivananda Poojara 52/34
-
References
• ppt 9-25 prepared by Chinmaya kumar Dehury
• https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-146.pdf
• www.buyya.com
• https://www.harbourit.com.au/blog/everything-you-need-to-know-about-hybrid-cloud/
• Buyya, R., Srirama, S. N., Casale, G., Calheiros, R., Simmhan, Y., Varghese, B., ... & Toosi, A. N. (2018). A manifesto for future generation cloud computing: Research directions for the next decade. ACM computing surveys (CSUR), 51(5), 1-38.
Shivananda Poojara 53/34
http://www.buyya.com/