cos301. = managed for youstandalone servers iaaspaassaas applications runtimes database operating...

Inside Windows Azure:The Cloud Operating System

Mark RussinovichTechnical FellowWindows Azure

COS301

Agenda

Introduction to Windows AzureWindows Azure FundamentalsFabric Controller InternalsDeploying a ServiceUpdating a ServiceHost OS UpgradesService Health

What is a “Cloud”?

Cloud: on-demand, scalable, multi-tenant, self-service compute resources

Cloud Fundamentals

Infrastructure as a Service (IaaS): basic compute and storage resources

On-demand serversAmazon EC2, VMWare vCloud

Platform as a Service (PaaS): cloud application infrastructureOn-demand application-hosting environmentE.g. Google AppEngine, Salesforce.com, Windows Azure

Software as a Service (SaaS): cloud applicationsOn-demand applicationsE.g. Office 365, GMail, Microsoft Office Web Companions

Cloud: Efficiency Versus Control

= Managed for You StandaloneServers

IaaS PaaS SaaS

Applications

Runtimes

Database

Operating System

Virtualization

Server

Storage

Networking

Windows Azure

Efficiency

Control

Windows AzureWindows Azure is an OS for the data center

Model: Treat the data center as a machineHandles resource management, provisioning, and monitoringManages application lifecycleAllows developers to concentrate on business logic

Provides shared pool of compute, disk and networkVirtualized storage, compute and networkIllusion of boundless resources

Provides common building blocks for distributed applicationsReliable queuing, simple structured storage, SQL storageApplication services like access control and connectivity

Windows Azure Application Philosophy: Design for Failure

Scale out for capacityScale out for redundancyShort time outs with retriesIdempotent operationsStateless with durable external storage

Windows Azure Application Characteristics

Automated, Consistent Application Updates

Updates to the application occur in an automated wayUpdates result in clean components forcing consistencyLocal storage and OS are left untouched

Automated, Consistent Configuration Changes

Updates to the settings occur in an automated wayUpdates result in clean settingsLocal storage and OS are left untouched

Multi-Instance Management

Identical instances are deployed across the serviceLarge scale-out services are guaranteed to be consistent No configuration drift

Scale-out Application scale-out can occur automatically

High Availability The application has no downtime, even in the face of hardware failures.

Automated, Consistent OS Servicing

The OS system hosting the application can be updated with the most recent patches in a coordinated and automated way.

Windows Azure Application Characteristics

Single InstancePersistent OS

Single InstanceStateless OS

Multi-InstanceStateless OS




Scale-out

High Availability


Windows Azure

Single InstancePersistent OS

Single InstanceStateless OS

Multi-InstanceStateless OS




Scale-out

High Availability


Windows Azure Platform Building BlocksWindows Azure ComputeWindows Azure Storage

BLOBsTablesQueues

Windows Azure CDNSQL AzureAppFabric PaaS Middleware Services

AppFabric CachingAppFabric Service BusAppFabric Access Control Server

Agenda


Modeling Cloud ApplicationsA cloud application is typically made up of different components

Front end: e.g. load-balanced stateless web serversMiddle worker tier: e.g. order processing, encodingBackend storage: e.g. SQL tables or filesMultiple instances of each for scalability and availability

Front-End

Mark’s Cloud Application

Front-End

HTTP/HTTPSWindows

AzureStorage,

SQL Azure

Load BalancerMiddle-

Tier

The Windows Azure Service Model

A Windows Azure application is called a “service”Definition informationConfiguration informationAt least one “role”

Roles are like DLLs in the service “process”Collection of code with an entry point that runs in its own virtual machine

Windows Azure compute SLA requires two instances of each role

99.95% for connectivity to two instancesAchieved with update and fault domains

Availability: Update DomainsPurpose: Ensure service stays up while updating and Windows Azure OS updatesSystem considers update domains when upgrading a service

1/Update domains = percent of service that will be offlineDefault and max is 5, but you can override with upgradeDomainCount service definition property

The Windows Azure SLA is based on at least two update domains and two role instances in each role

Front-End-1

Front-End-2

Update Domain 1

Update Domain 2

Middle Tier-1

Middle Tier-2

Middle Tier-3

Update Domain 3

Middle Tier-3

Front-End-2Front-End-1

Middle Tier-2Middle Tier-1

Availability: Fault Domains

Purpose: Avoid single points of failures

Similar concept to update domainsBut you don’t control the updates

Unit of failure based on data center topology

E.g. top-of-rack switch on a rack of machines

Windows Azure considers fault domains when allocating service roles

2 fault domains per serviceWill try and spread roles out across moreE.g. don’t put all roles in same rack

Front-End-1

Fault Domain 1

Fault Domain 2

Front-End-2

Middle Tier-2

Middle Tier-1

Fault Domain 3

Middle Tier-3

Role Contents

Definition: Role nameRole typeVM size (e.g. small, medium, etc.)Network endpoints

Code: Web/Worker Role: Hosted DLL and other executablesVM Role: VHD

Configuration:Number of instancesNumber of update and fault domains

Mark’s ServiceRole: Front-End

DefinitionType: WebVM Size: SmallEndpoints: External-1ConfigurationInstances: 2Update Domains: 2Fault Domains: 2

Role: Middle-Tier

DefinitionType: WorkerVM Size: LargeEndpoints: Internal-1ConfigurationInstances: 3Update Domains: 2Fault Domains: 2

Role Types

There are currently three role types:Web Role: IIS7 and ASP.NET in Windows Azure-supplied OSWorker Role: arbitrary code in Windows Azure-supplied OSVM Role: uploaded VHD with customer-supplied OS

VM Role: is it a VM?No, because it is statelessGood for:

Long install (5+ minutes)Manual install/configFragile install/config

Service Model Files

Service definition is in ServiceDefinition.csdefService configuration is in ServiceConfiguration.cscfgCSPack program Zips service binaries and definition into service package file (service.cscfg)

RDFEService

Portal Service

US-North Central Datacenter

Deploying a Service to the Cloud:The 10,000 foot view

Service package uploaded to portalWindows Azure Portal Service passes service package to “Red Dog Front End” (RDFE) Azure serviceRDFE converts service package to native “RD” version

RDFE sends service to Fabric Controller (FC) based on target regionFC stores image in repository and deploys and activates service

FC

Service

Agenda


The Fabric Controller (FC)

The “kernel” of the cloud operating systemManages datacenter hardwareManages Windows Azure services

Four main responsibilities:Datacenter resource allocationDatacenter resource provisioningService lifecycle managementService health management

Inputs:Description of the hardware and network resources it will controlService model and binaries for cloud applications

Server

Kernel

Process

Datacenter

Fabric Controller

Service

Windows Kernel

Server

WordSQL

Server

Fabric Controller

Datacenter

ExchangeOnline

SQL Azure

Sidebar: What’s with all these “Fabrics”?

The Windows Azure Fabric Controller is totally, completely, unrelated to AppFabricAppFabric is a brand that encompasses:

Windows Server AppFabric: a set of components for building composite applications based on Windows Communication Foundation and Windows WorkflowWindows Azure AppFabric: Cloud services for connecting cloud and on-premise applications

AppFabric Access Control Server AppFabric Service BusAppFabric CacheBuilt as Windows Azure services

Datacenter Architecture

TOR

LB LBAgg

PDU

LB LBAgg

LB LBAgg

LB LBAgg

LB LBAgg

LB LBAgg

Racks

Datacenter Routers

Aggregation Routers and

Load Balancers

TOR

PDU

TOR

PDU

TOR

PDU

TOR

PDU

TOR

PDU

TOR

PDU

TOR

PDU

TOR

PDU

TOR

PDU

TOR

PDU

TOR

PDU

TOR

PDU

TOR

PDU

TOR

PDU

……… … …

Top of RackSwitches

Power Distribution Units

…

Nodes

Nodes

Nodes

Nodes

Nodes

Nodes

Nodes

Nodes

Nodes

Nodes

Nodes

Nodes

Nodes

Nodes

Nodes

Windows Azure Datacenters

High-Level FC ArchitectureFC is a distributed, stateful application running on nodes (blades) spread across fault domains

Installed by “Utility” Fabric ControllerOne acts as the primary and all others keep view of world in syncSupports rolling upgrade, and services continue to run even if FC fails entirely

TOR

FC1

… …

TOR

FC2

… …

TOR

FC3

… …

FC3

TOR

FC4

… …

TOR

FC5

… …

LB

LB AGG LBLB LB

Nodes

Rack

Provisioning a Node

Power on nodePXE-boot Maintenance OSAgent formats disk and downloads Host OSHost OS boots, runs Sysprep /specialize, rebootsFC connects with the “Host Agent”

Fabric Controller

RoleImagesRole

ImagesRole

ImagesRole

Images

Image Repository

Maintenance OS

Parent OS

Node

PXEServer

Maintenance OS

Windows AzureOS

Windows Azure

OS

FC Host

Agent

Windows Azure Hypervisor

Inside a Node

Fabric Controller (Primary)

FC Host Agent(trusted)

Host Partition

Guest Partition

Guest Agent

Guest Partition

Guest Agent

Guest Partition

Guest Agent

Guest Partition

Guest Agent

Physical Node

Fabric Controller (Replica) Fabric Controller (Replica)…

Role Instance

Role Instance

Role Instance

Role Instance

Trust boundary

27

Agenda


Service Deployment StepsProcess service model files

Determine resource requirementsCreate role images

Allocate compute and network resourcesPrepare nodes

Place role images on nodesCreate virtual machinesStart virtual machines and roles

Configure networkingDynamic IP addresses (DIPs) assigned to bladesVirtual IP addresses (VIPs) + ports allocated and mapped to sets of DIPsPrograms load balancers to allow traffic

Service Resource Allocation

Goal: allocate service components to available resources while satisfying all hard constraints

HW requirements: CPU, Memory, Storage, NetFault domains

Secondary goal: Satisfy soft constraints Prefer allocations which will simplify servicing the host OS/hypervisor: pick nodes that already have instances from the same update domainOptimize network proximity: pack nodes

Service allocation produces the goal state for the resources assigned to the service components

Node and VM configuration (OS, hosting environment)Images and configuration files to deployProcesses to start

Service allocation also allocates network resources such as LB and VIPs

Example Service AllocationRole BCount: 2

Update Domains: 2Fault Domains: 2

Size: Medium

Role ACount: 3

Update Domains: 2Fault Domains: 2

Size: Large

Fault Domain 1 Fault Domain 2 Fault Domain 3

LoadBalancer

10.100.0.36

10.100.0.122

10.100.0.185

www.mycloudapp.net

www.mycloudapp.net

http://www.mycloudapp.net/

http://www.mycloudapp.net/

Provisioning a Role InstanceFC pushes role files and configuration information to target node host agentHost agent creates three VHDs:

Differencing VHD for OS image (D:\)Host agent injects FC guest agent into VHD for Web/Worker roles

Resource VHD for temporary files (C:\)Role VHD for role files (first available drive letter e.g. E:\, F:\)

Host agent creates VM, attaches VHDs, and starts VMGuest agent starts role host, which calls role entry point

Starts health heartbeat to and gets commands from host agent

Load balancer only routes to external endpoint when it responds to simple HTTP GET (LB probe)

Inside a Role VM

Resource Volume

OS Volume

Role Volume

Guest Agent

Role Host

Role Entry Point

Provisioning VM Role InstancesVM Role base and differencing VHD are stored in Windows Azure Storage blobs

Shadow versions are made when the originals are uploadedVHD reads all go through a VHD caching service

Reads come on-demand from the cacheWrites go to a secondary differencing VHD

“Reimage” simply deletes it and reboots

Windows Azure Blob StorageOriginal Base VHD

Original Differencing VHD

Shadow Base VHD

Shadow Differencing VHD

RDFE

VHD CachingService

Base VHD

Shadow Differencing VHD

Secondary Differencing VHD

Node

Agenda


Update TypesThere are two update types:

In-placeVIP swap

In-place update:Role instances upgraded one update domain at a timeTwo modes: automatic and manual

VIP swap update:New version of service deployed, external VIP/DIP mapping swapped with old

Role AUD 1

Role AUD 2

Role BUD 1

Role BUD 2

Role AUD 1

Role BUD 1

Role AUD 2

Role BUD 2

Role AUD 1

Role AUD 2

Role BUD 1

Role BUD 2

Role AUD 1

Role AUD 2

Role BUD 1

Role BUD 2

LB

In-Place Update

VIP Swap Update

Update Type Characteristics

Supported Changes In-place VIP Swap Delete and Redeploy

Operating system version Yes Yes Yes

.NET trust level Yes Yes Yes

Virtual machine size Soon Yes Yes

Local storage settings Yes

(increase only)

Yes Yes

Number of roles Soon Yes Yes

Number of role instances Yes Yes Yes

Number or type of endpoints Soon No Yes

Names and values of configuration settings Soon Yes Yes

Values (but not names) of configuration settings Yes Yes Yes

Add new certificates Soon Yes Yes

Change existing certificates Yes Yes Yes

Deploy new code Yes Yes Yes

In-Place Update Detail

FC deploys updated role files and configuration to all nodes in parallelPrepares new role instances:

FC host agent creates new role VHDAttaches and mounts new role VHD

Stops old role instance:FC instructs guest agent to stop role instanceDismounts and detaches old role VHD

Starts new role instances:Calls new role code entry pointConsiders role instance update successful when role code reports “ready”

Note that resource volume is preserved updates of role instance

Agenda


Updating the Host OSInitiated by the Windows Azure team

Typically no more than once per month

Goal: update all machines as quickly as possibleConstraint: must not violate service SLA

Service needs at least two update domains and role instances for SLACan’t allow more than one update domain of any service to be offline at a time

Note: your role instance keeps the same VM and VHDs, preserving cached data in the resource volumeEssentially a graph coloring problem

Edges exist between vertices (nodes) if the two nodes host instances of the same service role in different update domainsNodes that don’t have edges between them can update in parallel

Example Allocations• Both allocations are

valid from the services point of view• Allocation 1 allows for

2 nodes rebooting simultaneously

• Allocation 2 allows only one node to be down at any time

• Host OS upgrade rollout is 2x faster with allocation 1

Allocation 1

Allocation 2

Service ARole A-1

UD 1

Service BRole A-1

UD 1

Service ARole B-1

UD 1

Service BRole B-1

UD 1

Service ARole A-1

UD 2

Service BRole A-1

UD 2

Service ARole B-2

UD 2

Service BRole B-2

UD 2

Service ARole A-1

UD 1

Service ARole B-1

UD 1

Service BRole B-1

UD 1

Service ARole A-1

UD 2

Service ARole B-2

UD 2

Service BRole B-2

UD 2

Service BRole A-1

UD 2

Service BRole A-1

UD 1

Agenda


Load Balancer Operation

FC programs load balancers (LB) to “probe” guest agent (GA) every 15 seconds

If the guest misses two probes, the LB stops forwarding traffic

The role can report “busy” status to the GA GA stops responding to probes

LB keeps an idle connection open for 30sUse keep-alive commands if the connection needs to be open longer

Node Health IndexTimeouts vary depending on node state and operationBased on heartbeats, which are typically 15 seconds

Used for status and recovery Health state sampler resets the index on successful pollOnce the index falls below zero, FC attempts to heal nodeFor example, host agent timeout is 10 minutes

Worst-case reaction time is timeout interval + heartbeat interval

NodeHealthIndex

MissedHeartbeats

Heartbeat Interval

MissedHeartbeat

RecoveryInitiated

Heartbeat Timeout

HealthTimeout

Healthy

Node and Role Health Maintenance

FC maintains service availability by monitoring the software and hardware health

Based primarily on heartbeats Automatically “heals” affected roles

Problem How Detected Fabric Response

Role instance crashes FC guest agent monitors role termination FC restarts role

Guest VM or agent crashes FC host agent notices missing guest agent heartbeats

FC restarts VM and hosted role

Host OS or agent crashes FC notices missing host agent heartbeat Tries to recover nodeFC reallocates roles to other nodes

Detected node hardware issue Host agent informs FC FC migrates roles to other nodesMarks node “out for repair”

Guest Agent and Role Instance Heartbeats and Timeouts

25 min

GuestAgent

ConnectTimeout

Guest Agent Heartbeat

5s

RoleInstanceLaunch

Indefinite

RoleInstance

Start

RoleInstanceReady

(for updates only)

15 min

Role Instance Heartbeat

15s

Guest Agent Heartbeat Timeout

10 min

Role Instance “Unresponsive” Timeout

30s

Load Balancer Heartbeat

15s

Load BalancerTimeout

30s

Guest Agent

Role Instance

Moving a Role Instance (Service Healing)

Moving a role instance is similar to a service updateOn source node:

Role instances stoppedVMs stoppedNode reprovisioned

On destination node:Same steps as initial role instance deployment

Warning: Resource VHD is not moved

ConclusionPlatform as a Service is all about reducing management and operations overheadThe Windows Azure Fabric Controller is the foundation for Windows Azure’s PaaS

Provisions machinesDeploys servicesConfigures hardware for servicesMonitors service and hardware health

The Fabric Controller continues to evolve

Book Signing and My Last Session

Book signing @ Teched bookstore: 12:30-1:30Case of the Unexplained 2:45

Track Resources

Don’t forget to visit the Cloud Power area within the TLC (Blue Section) to see product demos and speak with experts about the Server & Cloud Platform solutions that help drive your business forward.

You can also find the latest information about our products at the following links:

Windows Azure - http://www.microsoft.com/windowsazure/

Microsoft System Center - http://www.microsoft.com/systemcenter/

Microsoft Forefront - http://www.microsoft.com/forefront/

Windows Server - http://www.microsoft.com/windowsserver/

Cloud Power - http://www.microsoft.com/cloud/

Private Cloud - http://www.microsoft.com/privatecloud/

http://www.microsoft.com/windowsazure/

http://www.microsoft.com/systemcenter/

http://www.microsoft.com/forefront/

http://www.microsoft.com/windowsserver/

http://www.microsoft.com/cloud/

http://www.microsoft.com/cloud/



Resources

www.microsoft.com/teched

Sessions On-Demand & Community Microsoft Certification & Training Resources

Resources for IT Professionals Resources for Developers

www.microsoft.com/learning

http://microsoft.com/technet http://microsoft.com/msdn

Learning

http://northamerica.msteched.com

Connect. Share. Discuss.

http://www.microsoft.com/teched

http://www.microsoft.com/learning

http://microsoft.com/technet

http://microsoft.com/msdn

http://northamerica.msteched.com/

Complete an evaluation on CommNet and enter to win!

Scan the Tag to evaluate this session now on myTech•Ed Mobile

© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to

be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS

PRESENTATION.

cos301. = managed for youstandalone servers iaaspaassaas applications runtimes database operating...

Documents

frontend slide

cos301 slide

consistent os servicing

server datacenter slide

agg pdu

middle tier

os system

update domains