topstack product architecture 2013-q3
Post on 17-May-2015
319 Views
Preview:
DESCRIPTION
TRANSCRIPT
TopStack Architecture
Q3 2013 Update
2
The basics
3
Overview
TopStack is a suite of services to extend Infrastructure as a Service (IaaS) solutions and deliver key Platform Services (PaaS)
TopStack delivers a clean-room implementation of many of Amazon’s most popular services
TopStack runs on private clouds as well as third party public clouds
The 2013-Q3 focus for TopStack is to act as a complement for OpenStack.
TopStack is available in both a Community Edition (open source) and an Enterprise Edition (commercial license & support).
4
Source Code Organization & Control
Source code for TopStack is stored in Git, open source is in github
Each service is stored in a separate repo
Common repo: ToughCore, for shared utility code
Common repo: ToughResources, for shared static assets
All repos have a master branch for current production code
Tag is applied for each production release
All repos have at least one development branch for current developmentAdditional feature branches are created for feature development, as needed
5
Build Management and Quality Assurance
Services are built with Ant build files, with Maven tasks for dependency resolution
Dependencies are resolved through local file copies (in dev mode)
Dependencies are resolved through Jenkins artifacts (in build mode)
Builds are managed through Jenkins continuous integration
All service include unit tests that run with each build
Services are deployed as part of continuous integration to Dev clouds
Post deploy, Java integration tests are performed against fresh deploy
Any failed integration tests will cause the build to be marked broken
Once a day, a “long running” set of integration tests are run
Long running tests spin up instances and test advanced connectivity
6
Continuous Deployment
Continuous deployment is performed by Jenkins, with jobs deploying to Dev, etc.
Deployments pushed to multiple cloud platforms, versions, …
Cloud 1 Cloud 2
7
Installation & Deployment
Installation package is a single package file (.tar.gz), output from continuous build
Unpacked, install package consists of:
Master installation shell script
Install guide (PDF)
Packaged services, to be deployed by installation script
Base Image configuration script
Installation may be re-run as needed to install/configure additional instances
Options to installation allow the installer to include/exclude particular services
Required supporting services are always installed
8
Deployment
Current tested deployment configuration:
OpenStack Grizzly or greater (older versions work, but are not commercially supported)
nova-compute with libvirt+KVM, libvirt+QEMU, libvirt+XenServer
nova-volume/Cinder, any iSCSCI backend
nova-network/Quantum, single VLAN
Linux VM, Ubuntu 12.04 or greater
9
Services Offered to Customers
Elastic Load Balancer
Route 53
Relational Database Service
ElastiCache
Simple Queue Service
CloudWatch
CloudFormation
Elastic Beanstalk
Auto Scale
This deck wont cover these in any
detail
10
Internal Services & Components
Internal Services (daemons):
Service registry & configuration
Orchestration & events
Job Scheduling
Common Components:
Common logging
Persistence
Instance configuration (Chef)
Authorization & access control
Quotas & metering
…
Cloud platform bindings
Instrumentation
Administration
Inter-Service Communication
11
DEPLOYMENT MODEL
12
Cloud Image Repo
TopStack Master VM
Tomcat 7
Deployment Model - Evaluation/TopStack Lite
12
TopStack SLB
DNS53
SQS
CloudWatch
RDS
Other TopStack Services
Apache 2
StackStudio
Chef Server
PubSub Queue
TS Base Image
TopStack DNS53
MySQL
13
TopStack SLB VM
TopStack Service VM2
Deployment Model - TopStack Enterprise
13
TopStack SLB
TopStack Service VM1
Tomcat 7
DNS53
SQS
CloudWatch
RDS
Other TopStack
StackStudio VM
Apache 2
StackStudio
Cloud Image Repo
Chef VM
Chef Server
[Optional]
DB VM
MySQL
Queue VM
PubSub Queue
DNS53 VM
DNS53
TS Base Image
14
Chef VM
TopStack Service VMn
Deployment Model - TopStack HA
14
TopStack Service VM2
StackStudio VM
Apache 2
StackStudio
TopStack Service VM1
TopStack ELB VM2TopStack ELB VM1
Cloud Image Repo
[Optional]
DB Active
MySQL
DB Standby
MySQL
Chef Cluster
Chef Server
Chef VMQueue Cluster
PubSub QueueChef VM
DNS53 Cluster
DNS53
TS Base Image
15
Internal Services
16
Service Registration & Configuration
All services must register with DNS53 on startup
DNS53 maintains private zone for Transcend internal use
Installation creates addresses for TopStack hosts
Registration creates CNAMEs for individual services in DNS
DNS information is used by Transcend load balancer to direct traffic
TopStack ServiceSLB
Request Handler Thread
17
Orchestration & Events
Request Handler Thread
Open Transaction
IaaS Provider
Create WF
ClientRequest
Response
Commit Transaction
TopStack Workflow
Cloud Op Task
Notify Task Complete
RDSWork
CFWork
SLBWork
Workflow Step 1
Workflow Error State
Quartz
Open Transaction
Commit Transaction
Cloud Op Task
Workflow Step 2
Rollback Resources
Continuation
Request ID Cache
18
Orchestration & Events
Services only own workflow steps and a light servlet for request/response
Pub-sub mechanism between TopStack API front end and service workers
ZeroMQ (http://www.zeromq.org)
Protocol Buffers as serialization format for ZeroMQ
Workflow solution to handle multiple asynchronous service steps:
Mule ESB (http://www.mulesoft.org/)
Asynchronous requests from HTTP handlers
Tomcat 7 with Servlet 3.0 asynchronous servlets (continuations)
Request IDs to marry asynchronous responses to requests
19
Workflow
19
Many services consist of multiple operations, both synchronous and asynchronous
For example, a Relational Database is created:
An instance must be spun up
Volume is created (in parallel)
Public IP must be associated
Instance startup is complete
Volume is attached
Database installation is performed
etc.
Any workflow step may fail, in which case:
Allocated resources must be torn down, freed
Failure must be reported, handled appropriately
20
Job Scheduling
Scheduled jobs are executed using Quartz Enterprise Job Scheduler
Quartz runs in clustered configuration
Jobs are executable by any TopStack instance
Scheduled jobs are stored in relational DB
Services may add new jobs to be executed during e.g. maintenance windows
Quartz is a source of workflow jobs
For example, on setting RDS maintenance window, a Quartz job is created
When Quartz job fires, RDS code is invoked to submit workflow
21
Common Components
22
Common Logging
Logging from all TopStack services is performed through SLF4J library
Logging implementation is typically Log4J
Logging may be directed to syslog (including TCP) or simple files
Configuration provides opportunity for aggregation, mining
23
Authorization & Access Control
Each TopStack account will require an active IaaS cloud credential set
IaaS credentials are encrypted at rest
Actions are performed using credentials associated with TopStack account
IaaS authorization and access limits define TopStack limits
24
Instance Configuration
Chef Server
Deployment includes an embedded Chef server (http://www.opscode.com/chef)
Embedded Chef includes a set of Transcend recipes to build up resources
Chef Client
Transcend Base Image burns a Chef client into the image
As new instances are started by TopStack, a Chef configuration and role are injected
Instances dial-back to TopStack as the final step of configuration to become ready
25
Persistence
Configuration and event data is stored in a relational database (default MySQL)
Data access is through a DAO layer and Hibernate, an O/R mapping layer
26
Cloud Platforms Bindings
TopStack configuration requires cloud “flavor” as input; OpenStack, Eucalyptus, etc.
IaaS cloud must provide the core operations used by TopStack (or equivalents):
Create/Terminate VM Instance
Allocate/Release IP Address
Associate/Disassociate IP Address
Describe Instances
Create/Delete Security Group
Describe Security Groups
Authorize/Revoke Security Ingress
Create/Delete Volume
Describe Volume
27
Quotas & Metering
All quotas enforced by IaaS provider apply to TopStack instances as well
Some quota is consumed by TopStack constructs that map to quota items
E.g., RDS security group consumes an IaaS security group
28
Instrumentation
All TopStack hosts are monitored as CloudWatch instances
Installation process configures hosts
Metrics are available though normal CloudWatch APIs
All TopStack service hosts expose basic management information
All hosted services are available, with service status
Service workers (workflow steps) maintain “health” information
Count of tasks processed
Count of task with abnormal outcome
Transactions processed per second
Collected via metrics (http://metrics.codahale.com/)
29
Administration
TopStack Enterprise Edition provides an Administration Console
Console runs on each TopStack host
Allows central administration of services
Allows provisioning of user accounts
Provides information on active services, failure rates, scheduled jobs
30
Inter-Service Communication
TopStack services communicate with each other only as workflow steps
Subsequent workflow steps are routed through Pub/Sub queue
Loosely coupled, via workflow
31
Non-functional requirements
32
High Availability
TopStack service hosts run in parallel on different VMs; scale-out architecture
VMs may be removed from service & load will redistribute across remaining instances
Workflows in progress will be continued by other instances
TopStack persistence tier may be run in master/slave or cluster configuration
33
Scalability
TopStack host machines can run any or all TopStack services
TopStack endpoints are load balanced across available service hosts
Many service hosts can run in an environment; new hosts register services on start
TopStack persistence tier scales vertically to support large transaction volumes
34
Portability
The Dasein cross-cloud library allows TopStack to operate against the most popular clouds
TopStack assumes only core IaaS services are available
Most clouds provide core IaaS services, or services which may be mapped to IaaS
35
Security
TopStack services are secured with access key and a secret key/password
Optionally, customer can add HSM for increased security
Secret key/password is not transmitted without encryption
Enterprise Edition provides additional OS level lock-downs (PCI DSS)
top related