mistral hong kong unconference track
DESCRIPTION
Mistral Unconference slides from Hong Kong OpenStack Design SummitTRANSCRIPT
MistralTask Orchestration and Scheduling service
• Home: launchpad.net/mistral
• Wiki: wiki.openstack.org/wiki/Mistral
• IRC: #openstack-mistral
• Me: Renat Akhmerov <[email protected]>
Background
• Realized by Murano team as a generic external state machine with HA and scalability
• Discussed with and supported by the authors of the Convection proposal, Joshua Harlow and TaskFlow team
• Joshua, Thanks!
How is Mistral related to TaskFlow & Convection
• Mistral implements the Convection (Workflow as a Service) and goes far beyond the initial ideas
• TaskFlow library will be used for implementation
• TaskFlow library will have Mistral engine to execute tasks over Mistral in a distributed manner
Mistral Mission• Build a Task Orchestration and Scheduling
service for OpenStack
• Provide an easy and flexible mechanism for executing workflows that consist of interrelated tasks in a cloud environment
• Help other OpenStack projects automate their internal workflows
• Provide HA and Scalability for task/workflow execution
Mistral Users• Infrastructure OpenStack developers:
Nova, Heat, Murano, Solum etc.
• System administrators: to use capabilities like Cloud Cron to be able to schedule helper cloud-wide tasks (local VM processes, RESTful services, creating/terminating VMs, etc.)
• Enterprise application developers: automate business processes consisting of multiple distributed processing steps
Reasons to use Mistral
1. High Availability: ability to continue a workflow from the point it crashed at
2. Scalability: Mistral knows what parts of a workflow can be executed in parallel
3. Scheduling: workflows/tasks can be scheduled for periodical execution (e.g. every Tuesday at 3 am)
4. Observable state: can always see the details of the current execution state and history
5. Offloading dependency management: given what needs to be completed Mistral executes whatever tasks needed for that
6. Additional integration points: a 3rd party application can hook into deployment processes represented as Mistral workflows using events
7. Formalized task graphs are just easier to manage and understand: they can be visualized, analyzed and optimized
Use Cases• Cloud Cron: a system administrator can
schedule cloud tasks for periodical execution
• Cloud Environment Deployment: a deployment tool or a system can represent deployment steps as a Mistral workflow and use Heat for every step as a software orchestration tool
• Event Based Scheduling: e.g. Live Migration on CPU 100% from Ceilometer
Cloud Cron
• Single point of configuration and control
• Wide range of tasks: local processes, RESTful services etc.
• Monitoring & Management
• HA & Scalability
Cloud Environment Deployment
• Formalized deployment workflow
• Individual tasks use Heat to install applications
• Dependency Management offloaded to Mistral
• Multiple flows in parallel
• Mistral provides HA and scalability
• No need to use locking
Event Based Scheduling
• Live migration is an example
• Workflow execution on external events
• Various event types
• Various event sources
Key Mistral Features
• Uploading custom graphs of tasks
• Driven by simple DSL: YAML/JSON/XML
• Task Scheduling: using UI or Cron patterns
• Generic service: doesn’t do the actual work itself and acts as an HA coordinator signaling to workers about what needs to be done
Main Concepts• Task Graph - a set of related
tasks
• Workflow - a subgraph involved into one execution
• Flow - a path that can be processed independently
• Target task - a task that needs to get completed in a particular case
• Action - a particular piece of work associated with a task (signal)
DSL# Tasks
tasks: Task1: action: action1
Task2: action: action2
Task3: requires: Task1, Task2 action: action3
# Actions
actions: action1: transport: rest1
action2: transport: amqp1
action3: transport: poll
# Transports
transports: rest1: type: REST url: http://some.url
amqp1: type: AMQP host: my.host.org port: 5672 routing-key: a_queue …
poll: type: POLL
Data Flow (Concept)
1. Select a subset of input data
2. Process it
3. Merge it back in
4. Pass it on
5. Merge with parallel branches
What about Mistral workers?
• Core engine doesn’t start any workers
• Workers are dynamic, configured in DSL
• Types of workers:• Client application nodes• REST services• Other processes accessible by AMQP,
SSH etc.
• BUT!
• Dedicated workers can be allocated using toolsets created on top of Mistral core engine
Can Mistral do anything else?
• Easy to build additional toolsets and frameworks
• Potential capabilities include:• Domains (Namespaces)• Event subscription• Role Based Access Control• Task priorities• Task collocation• Language-oriented frameworks (Python, Java, etc.)• Plugin system to introduce new DSL keywords
Is Mistral similar to Amazon Simple Workflow
• In many ways Mistral is similar to Amazon SWF
• Amazon SWF is oriented to language bindings (Java, Ruby, Python)
• Conceptual differences in underlying model
• Mistral is targeting to be much simpler
• Mistral may later have Amazon SWF adapter
Current Status
• Documented all the core ideas and concepts
• Prepared a draft of DSL/API specification
• Planned the Roadmap
• Started prototyping core ideas
• Introduced at HK OpenStack design summit! :)
Roadmap• Nov 25th, 2013.
• DSL/API v1.0 specifications
• Dec 10th, 2013.
• Mistral PoC: Basic task orchestration, DSL, API and Scheduling
• Jan 20th, 2014. Release 0.1
• DSL v1.0 and API v1.0 implementation
• Feb 20th, 2014. Release 0.2:
• Task engine HA & scalability, Web UI for monitoring & management
• March 20th, 2014. Release 0.3:
• Triggers (Cron, Events), Web UI for triggers (Cron, Events)
Thanks!
Q & A