one click cloud orchestrator (occo): bringing complex complete infrastructures effortlessly to iaas...
DESCRIPTION
BOINC project server BOINC client The whole infrastructure should be created by 1 click (Number of BOINC clients is a parameter) BOINC client Build an institutional desktop grid for universities Required BOINC expertise Took several weeks to establish Goal: setup the desktop grid in ~10 minutes in the cloud Easy creation of institutional desktop grids Easy creation of institutional desktop grids Used in IDGF-SP project Used in IDGF-SP project Motivations for OCCO (Example 1) Univ. cloud (or EGI cloud)TRANSCRIPT
One Click Cloud Orchestrator (OCCO): bringing Complex Complete
Infrastructures Effortlessly to IaaS Clouds
Peter KACSUK, Adam VISEGRADI, Jozsef KOVACS andMárk GERGELY
MTA SZTAKI
Common national pools of resources
Developing an Open Science Backbone
Developing an Open Science Commons
Governan
ce
Operatio
ns
Securit
y
Shared capabilities based on open standards
Research Infrastructures and long tail of science
Core capabilities• Open Science
Cloud (e.g., VM management, Data storage/access/discovery)
• PID• Service registry
and marketplace
• Research platform built on top of shared capabilities plus community owned resources
• Data products, tools, scientific gateways, virtual labs
Multi-level governance with community participation• Local• National• European
Federated operations and support• Service desk• Monitoring and
accounting• Capacity management• Service level
management
• Network of CSIRT• Federated IdPs,
Auth and Authz• Management of
different levels of assurance
From Member States• Capacity dedicated to
large RIs• Free pools for long tail
researchers• Both pubicly funded
and commercial providers (all supporting open standards and no lock-in)
Credit to Sergio Andreozzi
BOINCproject server
BOINCclient
The whole infrastructure should be created by 1 click(Number of BOINC
clients is a parameter)
BOINCclient
• Build an institutional desktop grid for universities
• Required BOINC expertise
• Took several weeks to establish
• Goal: setup the desktop grid in ~10 minutes in the cloud
• Easy creation of institutional desktop grids
• Used in IDGF-SP project
Motivations for OCCO (Example 1)
Univ. cloud (or EGI cloud)
SZTAKI Desktop Grid server
Gateway(WS-PGRADE)
To run PS applications on BOINC
3G Bridge BOINCproject
DC-API plugin
Metajob plugin
BOINCclient
Virtualisationby
GBACGBAC
BOINCclient
Virtualisationby
GBAC
Univ. cloud or EGI cloud
Build the BOINC infrastructure with a gateway to submit parameter sweep applications into BOINC
Motivations for OCCO (Example 2)
SZTAKI Desktop Grid server
Autodock Gateway 3G Bridge BOINC
project
DC-API plugin
Metajob plugin
BOINCclient
Virtualisationby
GBACGBAC
BOINCclient
Virtualisationby
GBAC
Univ. cloud or EGI cloud
Setting up an Autodock SaaS (gateway with BOINC infrastructure):
Motivations for OCCO (Example 3)
Biologist, chemist can use it without any learning curve
Using Autodock SaaS service
E-mail notification of the user
Admin page showing the setup of the service
E-mail notification of the user
Motivations for OCCO
• The creation of this 1-click infrastructure took several weeks -> we want a solution that enables the construction in several days
• There are many other use cases where the goal is to set up a complete infrastructure on demand in the cloud
• Such an infrastructure typically consists of multiple virtual machine applications• These should be instantiated and managed in an automated way• We need an offline description how such an infrastructure should look like• Then the infrastructure can be instantiated automatically, or with a single click of a
user. Hence the name: One-Click Cloud Orchestrator• The infrastructure typically defined as a graph where nodes are infrastructure
services.
• Automation possibilities in a cloudo Node instantiation
• Through an API or a UIo Node configuration management
• Via Chef, Puppet, etc.o Our goal is to extend these to infrastructure
Problems to be solved
• Complex, multi virtual machine applications need special care that are not offered by current infrastructure cloudso Users frequently must fine tune their virtual infrastructures manually to meet
their applications needs (scaling, inter-vm dependencies, error resilience)o Users of multiple clouds are usually restricted to the use of a particular provider
• Users of complex, multi virtual machine applications currently need low level understanding of application subcomponents and clouds for:o Configuration and deployment of subcomponents in a cloud context (this is
partially offered by configuration management tools like Chef or Puppet)o Such VMs must be crafted for the subcomponents that are capable to interface
with other dynamically created VMs
• We are developing a tool for the EGI communities to provide easy-to-use management for multi VM applications
State of the Art
feature\System Juju OneFlow Cloud- Formation Heat SlipStream OpsWorks"OneClick" plans "OneClick" plans
"OneClick" “vision”31-Jul-14 31-Jul-15
OS Support Ubuntu Hypervisor dependantFixed list of Linux
distributionsWindows Server
FreeBSDHypervisor dependant Supports wide variety
of OS
Amazon Linux,Supports wide variety
of OSSupports wide variety
of OSSupports wide variety
of OS
Ubuntu 12.04 LTS
Supported cloud backends
AWS EC2, HP Cloud Services, Windows Azure, Openstack
OpenNebula AWS EC2 OpenStackOpenStack, AWS EC2, OCCI, Microsoft Azure,
etc.AWS EC2
General EC2, General EC2,
All widely accepted Cloud Interfaces
could be easily extended later OCCI,
could be easily extended later
Node management method Service Composer Image Based Service Composer Service Composer Image Based Service Composer
Service Composer, Service Composer, Chef, other SotA
composer(s)
Generic Service Composer, Integrated
Abstract image managementChef
Open source Yes Yes No Yes Partly No Yes (?) Yes (?) Yes (?) Hosted service Hosted No Hosted No Hosted Hosted No No No + SaaS
Infrastructure Auto healing No Yes Yes Yes No Yes Hopefully Yes Yes
Manual Infrastructure Scalability Yes Yes Yes Yes Yes Yes No Yes Yes
Automatic Infrastructure Scaling No Yes Yes Yes No Yes No Yes Yes
UI CLI, GUI Web frontend, CLI management console, CLI, API
CLI, API, Horizon dashboard Web UI management console,
CLI, SDK API, CLI API, CLI, HTML5 Web frontend
API, CLI, HTML5 Web frontend, SaaS
One Click UI No No No No No No Yes Yes Yes
Service composer support Juju — Chef, Puppet Chef, Puppet — Chef loose integration with
Chefloose integration with
Chef, other SotA composer(s)
Generic Service Composer, Integrated
Abstract image management
Multi-cloud provisioning Yes No No No Yes No Yes Yes Yes
Maturity (subjectively between 1-5) 3 1 5 2 2 1 1 2 4
OCCO in general
• Instead of basic components like virtual machines, OCCO can provide complete infrastructures of services in an on-demand, self-service fashion
• Automatic scaling and error recovery will be included.• OCCO aims both providing
o a high-level service with UI o exposing the underlying architecture as a framework to be built upon
• OCCO orchestrates o resource provider backends (e.g. IaaS clouds) o and configuration manager backends (e.g. Chef) based on statically defined infrastructure descriptions (“infrastructure-as-code” paradigm).
• The OCCO framework is developed to be highly versatile and backend-agnostic.
One Click Cloud Orchestrator (OCCO)
• Analogous to IaaS clouds: users can simply request complex infrastructures via a simple UI/API.o Instead of choosing from virtual machine types, users choose from virtual
infrastructure typeso They still face the same pay as you go model, and they can still utilize the
flexibility of the IaaS’s just on larger scale.
OCCO
Node resolution
OCCO
16
View of the OCCO Administrator
• OCCO Administrators are expected to create node definitions and infrastructure descriptions (templates)o Based on existing configuration management descriptions (e.g. Chef recipes)o The templates are stored inside the OCCO provided template store in order to
allow reuse
17
View of the OCCO User
• Can browse and customize offered infrastructure description templates• Can receive notifications about the state of his/her infrastructure (to be
implemented)o Through email or an automated service (allowing immediate use after creation or
changes)o If a change is applied to the template then the new updated deployment descriptor can
be pushed to OCCO internals
Internal Architecture
18
• Automated Infrastructure maintenanceo infrastructure descriptor processing and
VM management initiator
• Infrastructure Processoro internal depiction of a virtual
infrastructure
• Cloud Handlero abstracts IaaS functionality (e.g., VM
creation) for federated and interoperable use of clouds
• Service Composer (vm reshaper)o ensures awaited functionalities for VMs
• Information Dispatchero de-couples the information producer and
consumer roles across the architecture
Definition of an infrastructure
Definition of an infrastructure
VM1 Cloud-init
Deployment example: Wordpress-mysql
MySQL
CHEF
OCCO
VM2Cloud-init
Wordpre
ss
EC2
Register MySql
Register Wordpress
Deploy Wordpress
DeployMySQL
Instantiate new VM with MySQL ipaddress
Instantiate new VM
Get MySQL ipaddress
Infrastructure description
Development environment of OCCO
• In alpha stage• Python 2.7• Git [https://gitlab.lpds.sztaki.hu/groups/cloud-orchestrator]• Jira [https://jira.lpds.sztaki.hu/browse/OCD] – agile sprint dev.• Documentation [http://c153-33.localcloud/util-doc/util.html]
o Sphinx, docstrings: automatic documentation generation• Testing: Python nosetests: automatic testing• Deployment: setuptools
o Development package repo: [http://c153-86.localcloud:8080/packages]o Works best with virtualenv
• Package dependencieso PyYAML, argparse, python-dateutil, pika (for RabbitMQ), Boto, etc.
• Will be open-source after first release in 2015 Q3
Documentation of OCCO
Conclusions
• Prototype version of OCCO is working• First version of OCCO is close to be released• Short term goal: to create OCCI plugin for EGI
FedCloud • New use cases are continuously developed• We are looking for partners to realize their use cases