Open Source Cloud

Kailash S

Centre for Development of Advanced Computing

(CDAC)

Chennai

CLOUD

CDAC Chennai India’s National Grid Computing Initiative

Advanced Computing Training School

National Resource Centre for Free and Open Source Software

Research & Development on Cloud

National Ubiquitous Computing Research Centre

MEGHDOOT A CDAC CLOUD INITIATIVE

For establishing cloud, develop & deploy applications in cloud

Enhanced Open Source Cloud Middleware

Offers IaaS,PaaS,SaaS

End to End Security Solutions

Certificate Programs in Cloud Computing

VALUE OF CLOUD

CLOUD EVOLUTION

GOVERNANCE - HYPECYCLE

Common Factors• Pay per use

• Instant Scalability

• Security

• Reliability

• APIs

IaaSPaaSSaaS

Advantages• Lower cost of ownership

• Reduce infrastructure management responsibility

• Allow for unexpected resource loads

• Faster application rollout

Cloud Economics• Multi-tenented

• Virtualisation lowers costs by increasing utilisation

• Economies of scale afforded by technology

• Automated update policy

Risks• Security

• Downtime

• Access

• Dependency

• Interoperability

CORPORATE FINDINGS

• Ten years ago, it could take three-to-six weeks of meetings, emails, hardware evaluations, and pricing negotiations just to get a set of servers provisioned for a development group, By now, using standard configurations and automated scripts in a virtual environment, CIOs could get those same servers up in either a public or private cloud in three-to-four hours. - Mark Settle, chief information officer for BMC Software

• Cloud computing, "saves us millions of dollars over five years over any of the other alternatives we looked at and provides us with worldwide data recovery, unprecedented integration and ease of use, and device independence.“According to Todd Pierce, Genentech's vice president of corporate information technology

• “We really don’t want to operate datacenters anymore. We’d rather spend our time giving our customers great service and writing great software than managing physical hardware.” —Don MacAskill, CEO, SmugMug

SURVEY

• According to a survey by Gartner, "Nearly 90 percent of organizations

surveyed expect to maintain or grow their usage of software as a service

(SaaS), citing cost-effectiveness and ease/speed of deployment as

primary reasons for adoption.“

• A research study, the Pew Internet & American Life Project, confirms that

“Younger employees are quickly adopting cloud-based applications:

about three fourths of Internet users in the 18-29 age bracket have used

webmail services”.

• IDC surveyed 244 enterprise users and concluded that “Cloud computing

will cross the chasm - move from early adopters to mainstream - over

the next three years.”

SURVEY OF 250 SMB

• Midmarket and enterprise organizations worldwide by Information Technology Intelligence Consulting (ITIC), Boston

• Companies are very clear on the technology benefits achieve by deploying a private cloud infrastructure.

– Ease of management (45 percent);

– Ability to provide better services to end users, customers and business partners (40 percent);

– Ubiquitous connectivity and anywhere access (40 percent); lower operational expenses (39 percent);

– Scalability as the business needs grow (38 percent) and lower upfront investment costs (36 percent).

PREDICTIONS

• “By 2012, early technology adopters will forgo capital expenditures and instead purchase 40 percent of their IT infrastructure as a service,” states Gartner Group

• By 2013, 12 percent of world software market will be Internet based forms of SaaS and cloud computing, according to Merrill Lynch

• By 2012, India-centric IT services companies will represent 20 percent of the leading cloud aggregators in the market (through cloud service offerings).

• Gartner is seeing India-centric IT services companies leveraging established market positions and levels of trust to explore nonlinear revenue growth models (which are not directly correlated to labor-based growth) and working on interesting research and development (R&D) efforts, especially in the area of cloud computing

• Cloud computing could create between 300,000 and 1.5 million new jobs in five years in Europe alone.” Professor Federico Etro, University of Milan, 2009. If so, that could reduce Europe’s unemployment rate from 0.3% to 0.6% and boost GDP growth by 0.1 - 0.3%.

COMPANIES MOST LIKELY IMPACTED

USE CASE

• The New York Times needed to convert 11 million articles and images in

its archive (from 1851 to 1980) to PDF. Their Internal IT said it would take

them seven weeks. In the meantime, one developer using 100 Amazon

EC2 simple Web service interface instances running Hadoop completed

the job in 24 hours for less than $300

• The U.S. Olympic Committee: using AT&T services to handle a busy traffic

during the games

• Microsoft Azure – T20

• Online Tamil Dailys

USERS

• Federal Chief Information Officer (CIO), Vivek Kundra, established the Cloud

Computing Initiative to fulfill the President's objectives for cloud computing.

• Defense Information Systems Agency (DISA)

• US Department of Energy (DOE)

• General Services Administration (GSA)

• United Kingdom – (G-Cloud)

• European Union - Seventh Framework Programme (FP7)

• The Digital Japan Creation Project (ICT Hatoyama Plan) - The Kasumigaseki Cloud

• Government of Victoria, Australia (Gov x.0)

• City Council of Biel, Switzerland

• Papa Murphy’s Take ‘N’ Bake Pizza

PREDICTIONS

95 billion

Market :

42 billon IDC

Merryl Linch

At this moment, the 5

major search engines

together have 2.000.000

computers

33% of IT business will be in

Cloud Computing

Gartner

Microsoft data centre in Chicago:

610.000 servers

8

8

80% fortune companies willpay to use cloud computingservices and 30% will pay forinfrastructure. Gartner5

RECOMMENDATIONS FOR SMB AND MID-MARKET ENTERPRISES

• Decisions– What data and processes to move to the Clouds

– What security you need to apply, you may be clear that

– What level you want to operate in the Clouds

– Which Cloud Formations are best suited to your needs.

• Evaluate Cloud– Service Level Agreements (SLAs) and contract terms.

– Address data security concerns upfront

– Application customization requirements.

– Invest more upfront in the evaluation and selection process

REASONS TO CONSIDER AVOIDINGCLOUD COMPUTING

• Security

• Data Location & Privacy

• Internet Dependency

• Performance & Latency

• Availability & Service Levels

• Current Enterprise Applications Can't Be Migrated Easily

• Reliability, Interoperability

• Vendor Lock in

• Lack of Standards

• Data transfer Bottlenecks

• Software licensing

ARCHITECTURAL CONSIDERATIONS

• Key Business Architectural Principles – Business Alignment, Cost Optimization

– Compliance with Laws and Regulations

– Business Agility

– Minimize Cost

• Key Application Architectural Principles – Technology Independence, Adherence to Standards

– Common Development Methodology

– Loosely coupled Interfaces

ARCHITECTURAL CONSIDERATIONS

• Key Information Architectural Principles – Implement Information Lifecycle Management

– Regulatory and Legal Compliance

– Enforce Data Privacy

• Key Technology Architectural Principles – Control Technical Diversity

– Adherence to Standards

– Scale Capacity and availability to satisfy Business Objectives

– Virtualize dependencies to hardware and software

– Unified Security Infrastructure

MEGHDOOTH

A CDAC Cloud Initiative

OPEN SORUCE CLOUD

• Free and open source cloud stack.

• One stop solution for implementing Cloud environment.

• Tools are compiled by enhancing the stack with value added components

developed by CDAC.

• Elasticity, provision for Windows hosting, Monitoring, Metering & Billing,

Web service based management, High Availability, Enhanced Security

across layers.

MEGHDOOTH

OFFERINGS OF MEGHDOOTH

• Private cloud offering platform and infrastructure as a service (PaaS and

IaaS).

• On demand dynamic provisioning

• Metering & Monitoring

• Graphical Installation component of Middleware; Web based

Management of Cloud resources

• One single middleware bundle that incorporates all pre-requisites

• API for easy deployment of multi instance user appliances

22

MEGHDOOT- a CDAC Cloud Initiative

Service Management

SaaS

PaaS

IaaS

Security

Monitoring & Maintenance

• Virtualized Computational & Storage Volumes

• Network – DNS, DHCP, NIS, VLAN, VPN, Load balancers, Bandwidth Provisioning

• VM, Storage, Network Management

• CDN, SDN• Instant VM Launching• Live VM Migration

• Elasticity for LINUX and WINDOWS

• Distributed Caching• Multi instances• Registry

Management• Data Management

• Multi tenancy• RBAC• Workflow

Management

• Reports & Alerts• Auditing & Accounting• Client interaction

through Discussion forums, blogs, social networking

• Physical Security• IDS & IPS• Firewall & Network Privacy• Data Privacy• IP TABLES & Security Groups• PKI & Encryption• Web Application Firewall• Monitoring console for security audit

• User Profile Management• Operation, Configuration,

Image Management• Servers, Storage & Network

Management• Metering & Billing• Service Provisioning• Backup & Disaster recovery• SLA Management & QoS• Integration &

Interoperability• Graphical interface for

Administration• REST SOAP APIs• Snapshot Management

FEATURES AND FUNCTIONALITIES OF MEGHDOOTH

VALUE ADDITIONS TO OPEN SOURCE TOOLS

• Elasticity component integrated with Eucalyptus,

• Provision for hosting windows instances,

• High Availability to the cloud environment,

• Web Service Interface for management of Cloud

• Web Application Firewall,

• E-Mail and SMS Gateway integration,

• Provision for deploying user licensed software

• Metering and billing solutions

• Self servicing portal

25

• Consultancy for Government of Kerala – Kerala State IT Mission + IIITM-K + CDAC Chennai.

• Pilot project : Hosting SPARK Application into cloud.

• Scenario

• Each department have individual server.

• Payroll server achieves peak load during Month end.

• Education department achieves peak load once in year (Results publication).

• Each department strives for additional resources during peak load and idle for rest of the time.

• Approach & Achievements

• All the applications can be hosted onto cloud.

• The servers can be shared by all departments.

• Optimal utilization of the resources.

• Applications enjoy the benefit of elasticity.

• Service availability even during peak load.

CONSULTANCY SERVICES

• Offer corporate training for various organizations

• “Certificate Course in Cloud computing” program in CDAC Chennai.

• Corporate Training on Cloud Computing

Training Services

E-GOV APPLICATION OVER CLOUD

Customized for universities

Open Source Cloud Tools

OPEN SOURCE IN CLOUD

• Open Source with appropriate license

• Open Design & Open Development

• Open Community & Open API

• Licenses permit and encourage redistribution leads to R&D

• Architecture enables programs be used as components where-ever possible

• Open-standards, interoperability, flexibility

• Multi-lingual support & Data mobility

• Lowered barriers to adoption

• No Vendor Lock-in

• Extensible

• Cheap Services

• Data mobility

• Packaging, installation, maintenance

• Eucalyptus - University of California• Nimbus - Globus alliance• Open Nebula - DSA Research, Spain• Hadoop - Apache• Cloud-era - on top of Hadoop• Reservoir - European Union FP7• OpenQRM - IaaS• Cloudloop - Open-Source Cloud Storage API and Management•Jboss - Red Hat API for clouds•UEC - Ubuntu Enterprise Cloud•Enomaly - Strong Multi-Tenancy & Security•Appscale - RACELab at UC Santa Barbara•Openstack - Rackspace & NASA, Citrix, Dell, AMD, Intel

VARIOUS CLOUD MIDDLEWARE

CLOUD OSS

–IaaS

•Eucalyptus, OpenNebula, Nimbus, Enomaly, OpenQRM, Puppet, Reservoir, Pupet, UEC, ControlTier

–PaaS

•Appscale, Wavwmaker, Gearman, Heroku

–SaaS

•eyeOS, TioLive, Guacamol, Cornelios, Coadunation

–File systems

•HadoopDFS, GlutterFS, CloudStore, GDFS

–Hypervisor

•Xen, KVM, VirtualBox, OpenVZ, Qemu, Abiquo

CLOUD OSS

–Storage

•Cassandra, CouchDB, Flare, Memcached, MongoDB, XtreemFS

–Deployment/Admin/Monitoring/Test

•Chef, Puppet, ZenOSS, Bitnami, Bitrock, Capistrano, CDT, Etics 2, Fabric

–Data Processing

•Hadoop MapReduce, Pig, Zookeeper

–Libraries

•jCloud, libvirt, libcloud

OPENNEBULA

Origin:

Research project by Ignacio M. Llorente and Rubén S. Montero (nowmanaged by C12G Lab)

Core Technology:

C++, Ruby, JAVA and XMLRPC API

Features:

•Support multiple hypervisor Xen, KVM and VMware,

•Support for Microsoft Windows and Linux machine images

•Flexible requirement/rank matchmaker scheduler; and workload and

•resource-aware allocation policies

•Support for multiple hardware platforms (FibreChannel, iSCSI, NAS

•shared storage

•Public cloud offering by REST interfaces to your users

•Implementation of OGF OCCI and Amazon EC2

UEC

Origin :

Ubuntu Enterprise Cloud (UEC) is a package stack of applications from Canonical (best with an Ubuntu server).

Features:

•UEC includes a number of open source tools to manage theinfrastructure, including open source cloud software Eucalyptus.•KVM support

•Interface compatibility with EC2

•Simple installation and deployment

•Basic administrative tools for system management and user accounting

NIMBUS

Origin:

University of Chicago

Core Technology:

Java and Python

Features:

•Support for WSRF based or Amazon EC2 WSDL web service APIs

•Can be configured to use familiar schedulers like PBS or SGE

•self-configuring virtual clusters via contextualization

•Workspace - standalone site VM manager

•OGSA Based : Infrastructure services, Execution Management, Data

Services, Resource Management Services, Security Services, Self-

management Services, Information Services

•Per-client usage tracking and quota

OPENSTACK

Origin:

Project by Rackspace cloud & NASA, Intel & AMD

Core Technology:

Python

Features:

•Openstack Compute – for managing Virtual machines

•Openstack Object storage – for creating reduntant, scalable data

storage

•Openstack Imaging Service – for discovery, registration and

delivery services for virtual disk images.

•Hypervisor support includes ESX, Hyper-V, KVM, Xen, and

XenServer

EUCALYPTUS

Origin:Research project in the Computer Science Department at the

University of California, Santa Barbara.

Overview:

•Elastic Utility Computing Architecture Linking Your Programs To

Useful Systems.

•Provides a highly robust and scalable Infrastructure as a Service

solution.

•Compatible with Amazon AWS (EC2, S3, EBS) - SOAP and REST

interfaces

•Runs on multiple Linux distributions and supports Xen and KVM

hypervisors.

•Ability to configure multiple clusters, each with private internal

network addresses, into a single Cloud.

Cloud Standards &

Interoperability

The chairman of the Cloud Summit Executive 2008 conference reportedly

started the event by joking that he asked

20 people to define cloud computing, and got 22 different answers.

CONSUMER, CREATOR & PROVIDER

STANDARDS FOR ADMINISTRATORS

• Open Cloud Manifesto

– Statement of the principles for maintaining openness in cloudcomputing. Within two months of its announcement, 250organizations signed on as supporters

• Open Cloud Consortium

– Development of standards for cloud computing and to develop aframework for interoperability among various clouds

• Distributed Management Task Force

– Builds on existing standards for server hardware virtualization

• Open Virtualization Format

– Simplifies interoperability, security, and virtual machine life-cyclemanagement by describing an open, secure, portable, efficient, andextensible format for the packaging and distribution of one or morevirtual appliances, integrity checking of the virtual machines

STANDARDS FOR APPLICATION DEVELOPERS

• Ensure uniform, consistent, high-quality software solutions

• Standards for Messaging

• Simple Message Transfer Protocol, Post Office Protocol,

Internet Messaging Access Protocol

• Web Services

– Representational State Transfer (REST), Simple Object Access Protocol

• Standards for Security

– Security Assertion Markup Language (SAML), Open Authentication

(OAuth)

CLOUD COMPUTING STANDARDS ROADMAP

- National Institute of Standards and Technology

• NIST Cloud Computing Standards Roadmap Working Group• Inventory of Standards Relevant to Cloud Computing

• Present areas with standardization gaps include• SaaS functional interfaces• SaaS self-service• Management interfaces• PaaS functional interfaces• Business support / provisioning / configuration• Security and privacy.

• Current standard• Security auditing and compliance• Identity and access management• SaaS application specific data and metadata• Resource description and discovery.

NIST WORKING GROUPS FOR STANDARDS

• Provide a strategy and standards based guidance for the federal cloud computing

implementation effort

– Cloud Computing Reference Architecture and Taxonomy Working Group

– Cloud Computing Standards Acceleration to Jumpstart Adoption of Cloud Computing

– (SAJACC) Working Group

– Cloud Computing Security Working Group

– Cloud Computing Standards Roadmap Working Group

– Cloud Computing Target Business Use Cases Working Group

CLOUD COMPUTING STANDARDS FOR INTEROPERABILITY

• Functional Interface

• Management Interface

• Self-service IaaS management interface– Open Cloud Computing Interface (OCCI) from the Open Grid Forum -

standard IaaS resource management interface.

– The Cloud Data Management Interface (CDMI) standard - storage management & storage functional interface.

• PaaS Interface

• SaaS Interface

CLOUD COMPUTING STANDARDS FOR PORTABILITY

• Open Virtualization Format (OVF) - open standard for packaging and distributing virtual appliances

• Distributed Management Task Force (DMTF)– Digital signatures to ensure the integrity of the machine images

– Licensing information in the form of a machine-readable EULA (End User License Agreement) understandable by Images

• Workload Portability in the Cloud

• Internet protocols for service access - REST, SOAP, and XML

• Federated identity standards for service authentication -SAML and Oauth

• Standards for managing virtualized environments.

PRINCIPLES OF AN OPEN CLOUD –Open Cloud Manifesto

• Challenges to cloud adoption (security, integration, portability,interoperability, governance/management, metering/monitoring) areaddressed through open collaboration and the appropriate use of standards.

• Cloud providers must not use their market position to lock customers intotheir particular platforms and limit their choice of providers.

• When new standards (or adjustments to existing standards) are needed, wemust be judicious and pragmatic to avoid creating too many standards.

• Any community effort around the open cloud should be driven by customerneeds, not merely the technical needs of cloud providers, and should betested or verified against real customer requirements.

• Cloud computing standards organizations, advocacy groups, andcommunities should work together and stay coordinated, making sure thatefforts do not conflict or overlap.

CLOUD INTEROPERABILITY GOALS

• Moving virtual machines and workloads from one cloud compute service

to another

• Single sign-on for users who access multiple cloud services

• Ability to deploy and provision resources from multiple cloud services

with a single management tool

• Letting one application span multiple cloud services

• Allowing data exchange between clouds

• Letting a private cloud application seamlessly obtain resources from a

public cloud when excess capacity is needed

INTEROPERABILITY ELEMENTS OF A CLOUD PLATFORM

• Data Portability

– Securely move customers data in and out of cloud.

• Standards

– Should reuse existing and commonly used standards

• Ease of Migration & Deployment

– Migration path that preserves, in a secure way, existing investments in applications and IT resources.

• Developer Choice

– Enable developer choice in tools, languages and runtimes.

CLOUD SERVICE INTERFACES

• Management APIs;

• Data Exchange Formats;

• Federated Identity and Security Policy APIs;

• Resource Descriptions;

• Data Storage APIs.

USE-CASE OF INTEROPERABILITY

• Dynamic Operation Dispatch to IaaS Clouds

• Copy Data Objects between Cloud-Providers

• Cloud Burst From Data Center to Cloud

• Migrate a Queuing-Based Application

• Migrate (fully-stopped) VMs from one cloud-provider to another

Typical Commercial Terms of Service

• Availability• Remedies for Failure to Perform.• Data Preservation.• Legal Care of Subscriber Information.• Scheduled Outages & Force majeure events• SLA Changes & Negotiations.• Security• Service API Changes.• Licensed Software & Compliance.• Network dependency.• Workload locations are dynamically assigned and are thus hidden

from clients.• Risks from multi-tenancy.• Data import/export, and performance limitations.• Control & Visibility.

THE PRIVATE CLOUD SCENARIO

• Network Dependency.

• Workload locations are hidden from clients.

• Risks from multi-tenancy.

• Data import/export, and performance limitations.

• Potentially strong security from external threats.

• Modest-to-significant up-front costs to migrate into the cloud.

• Significant-to-high up-front costs to migrate into the cloud.

• Limited resources.

THE COMMUNITY CLOUD SCENARIO

• Highly variable up-front costs to migrate into the cloud.

• Limited resources.

• Network dependency.

• Workload locations are hidden from clients.

• Risks from multi-tenancy.

• Data import/export, and performance limitations .

• Potentially strong security from external threats.

• Modest-to-significant up-front costs to migrate into the cloud.

THE PUBLIC CLOUD SCENARIO

• Network dependency.

• Workload locations are hidden from clients.

• Risks from multi-tenancy.

• Limited visibility and control over data regarding security.

• Low up-front costs to migrate into the cloud.

• Elasticity: illusion of unlimited resource availability.

• Restrictive default service level agreements.

SOFTWARE-AS-A-SERVICE

• Very Modest Software Tool Footprint

• Efficient Use of Software Licenses

• Centralized Management and Data

• Platform Responsibilities Managed by Providers

• Isolation vs. Efficiency (Security vs. Cost Tradeoffs)

• Candidate Application Classes

• Data Protection.

• Client Device/Application Protection.

PLATFORM-AS-A-SERVICE CLOUD ENVIRONMENTS

• Abstract Interaction Dynamics

• Software Stack and Provider/Subscriber Scopes of Control

• Facilitated Scalable Application Development and Deployment

• Lack of Portability between PaaS Clouds

• Event-based Processor Scheduling

• Security Engineering of PaaS Applications

• Candidate Application Classes

• Generic Interfaces.

• Standard Languages and Tools.

• Data Access & Protection

INFRASTRUCTURE-AS-A-SERVICE CLOUD ENVIRONMENTS

• Abstract Interaction Dynamics

• Software Stack and Provider/Subscriber Scope of Control

• Full Control of the Computing Resource Through Administrative Access to VMs

• Flexible, Efficient Renting of Computing Hardware

• Portability, Interoperability with Legacy Applications

• Compatibility with Legacy Security Vulnerabilities

• Virtual Machine Sprawl

• Robustness of VM-level Isolation

• Features for Dynamic Network Configuration for Providing Isolation

• Multi-tenancy.

• VM Migration.

• Virtualization Best Practices.

OPEN ISSUES

• Computing Performance

• Latency

• Off-line Data Synchronization

• Scalable Programming

• Data Storage Management

• Cloud Reliability

• Network Dependence

• Cloud Provider Outages

• Safety-Critical Processing

ECONOMIC GOALS

• Risk of Business Continuity

• SLA Evaluation

• Portability of Workloads

• Interoperability between Cloud Providers

• Disaster Recovery

• Compliance

• Jurisdiction and Regulation

• Information Security

MANAGEMENT

• Migrating Data to and from Clouds.

• Continuity of Operations.

• Compliance.

• Administrator Staff.

• Legal.

• Operating Policies.

• Acceptable Use Policies.

• Licensing.

• Patch Management.

DATA GOVERNANCE

• Data Access Standards.

• Data Separation.

• Data Integrity.

• Data Regulations.

• Data Disposition.

• Data Recovery.

SECURITY AND RELIABILITY

• Subscriber-Side Vulnerabilities.

• Encryption.

• Physical.

• Authentication.

• Identity and Access Management.

• Performance Requirements.

• Visibility

SOFTWARE AND APPLICATIONS

• Time-critical Software.

• Safety-critical Software.

• Application Development Tools.

• Application Runtime Support.

• Application Configuration.

• Standard Programming Languages.