Cloud Cruiser and Microsoft Azure Usage API Integration

In this article:

Introduction

Microsoft Azure Tags

Creating a Resource Group with tags on Microsoft Azure

Import data from the Usage API into Cloud Cruiser

Processing the tag information from the Usage API

Adding the tag information to the consumption data

Introduction

Microsoft recently introduced the Azure Usage API, allowing subscribers to programmatically pull in usage

data to gain insights into their consumption. This is great news for Cloud Cruiser customers that can take

advantage of the richer dataset available through this API.

Cloud Cruiser can leverage the integration with the Usage API in several ways. The granularity (hourly usage

information) and resource metadata information available through the API provides the necessary dataset

to support flexible Showback or Chargeback models.

In this tutorial, we will present one example of how Cloud Cruiser can benefit from the Usage API

information. More specifically, we will create a new Resource Group on Azure, associate tags for the

account structure, and then describe the process of pulling and processing the tag information into Cloud

Cruiser.

The final goal is to be able to create reports like the one below, and be able to analyze cost and

consumption based on the account structure populated by the tags.

Figure 1 - Report with breakdowns using tags

Microsoft Azure Tags

The data available through the Azure Usage API includes not only consumption information, but also

resource metadata including any tags associated with it.

Tags provide an easy way to organize your resources, but in order to be effective, you need to ensure that:

- Tags are correctly applied to the resources at provision time;

- Tags are properly used on the Showback/Chargeback process to tie the usage to the

organization’s account structure.

Both of these requirements can be challenging, especially when there is some sort of manual process on

the provision or charging side. Mistyped, incorrect or even missing tags are common complaints from

customers when using tags and these errors can make life on the charging side very difficult.

With the new Azure Usage API, Cloud Cruiser can pull resource tagging information, and through a very

sophisticated ETL tool called workbooks, fix these common tagging errors. Through transformation steps

leveraging regular expressions and data correlation, Cloud Cruiser is able to identify incorrectly tagged

resources and apply the correct tags, filling the gaps and ensuring the correct association of the resources

with the consumer.

On the charging side, Cloud Cruiser automates the Showback/Chargeback process, and can leverage the

tag information to tie the usage to the appropriate consumer (Department, Division, Project, etc.). This

automation provides a huge improvement and can ensure a consistent and auditable charging process.

Creating a Resource Group with tags on Microsoft Azure

The first step in this tutorial is to create a new Resource Group on the Azure Portal and then create new

tags to associate to the resources. For this example, we will be creating the following tags:

Department

Environment

Owner

Project

The screenshot below of the Azure Portal shows a sample Resource Group with the associated tags.

Figure 2 - Resource Group with associated tags on Azure Portal

The next step is to pull the information from the Usage API into Cloud Cruiser. The Usage API currently

provides responses in JSON format. Here is a sample of the data retrieved:

{

"id": "/subscriptions/bb678b04-0e48-4b44-XXXX-

XXXXXXX/providers/Microsoft.Commerce/UsageAggregates/Daily_BRSDT_20150623_0000",

"name": "Daily_BRSDT_20150623_0000",

"type": "Microsoft.Commerce/UsageAggregate",

"properties": {

"subscriptionId": "bb678b04-0e48-4b44-XXXX-XXXXXXXXX",

"usageStartTime": "2015-06-22T00:00:00+00:00",

"usageEndTime": "2015-06-23T00:00:00+00:00",

"meterName": "Compute Hours",

"meterRegion": "",

"meterCategory": "Virtual Machines",

"meterSubCategory": "Standard_D1 VM (Non-Windows)",

"unit": "Hours",

"instanceData": "{\"Microsoft.Resources\":{\"resourceUri\":\"/subscriptions/bb678b04-0e48-4b44-XXXX-

XXXXXXXX/resourceGroups/DEMOUSAGEAPI/providers/Microsoft.Compute/virtualMachines/MyDockerVM\",\"locati

on\":\"eastus\",\"tags\":{\"Department\":\"Sales\",\"Project\":\"Demo Usage

API\",\"Environment\":\"Test\",\"Owner\":\"RSE\"},\"additionalInfo\":{\"ImageType\":\"Canonical\",\"ServiceType\":\"Sta

ndard_D1\"}}}",

"meterId": "e60caf26-9ba0-413d-a422-6141f58081d6",

"infoFields": {},

"quantity": 8

}

},

Import data from the Usage API into Cloud Cruiser

Cloud Cruiser workbooks provide an automated way to collect and process information from the Usage

API. An ETL (extract-transform-load) workbook allows you to configure the collection, transformation, and

publishing of data into the Cloud Cruiser database.

Each workbook can have one or multiple collections. This allows you to correlate information from different

sources to complement or augment the usage data. For this example, we will create a new sheet in the

Azure template workbook (UsageAPI) and set a new collection to import information from the Usage API.

Figure 3 - Usage API data imported into the UsageAPI sheet

Notice that this workbook already has other sheets to import services from Azure (ImportServices), and

process the consumption information from the Billing API (PublishData).

We are going to extract and process the information from the Usage API on the UsageAPI sheet, and

correlate the information with the consumption data from the Billing API on the PublishData sheet.

Processing the tag information from the Usage API

After importing the data into the workbook, we will create transformation steps in the UsageAPI sheet in

order to process the information from the API. First step is to use a “JSON split” processor to extract the

tags from a single field (as they are imported from the API) and create new fields for each one of them

(Department, Project, Owner and Environment).

Figure 4 - Create new fields for the tag information

Notice that the “Networking” service is missing the tag information (yellow box), but we can tell that this

service is part of the same Resource Group by looking at the ResourceGroupName field. Since we have the

tags for the other resources from this same Resource Group, we can use this information to apply the

missing tags to this resource later in the process.

Next step is to create a lookup table associating the information from the tags to the ResourceGroupName.

This lookup table will be used on the next step to enrich the consumption data with tag information.

Adding the tag information to the consumption data

Now we can jump to the PublishData sheet, which processes the consumption information from the Billing

API, and add the fields extracted from the tags. This process is performed by looking at the lookup table

created on the previous step, using the ResourceGroupName as the key for the lookups.

Figure 5 - Populating the account structure with the information from the lookups

Notice that the appropriate account structure fields for the “Networking” service were applied, fixing the

issue with the missing tags. We also populated the account structure fields for resources other than our

target Resource Group with “Other”, in order to differentiate them on the reports.

Now we just need to add another step to publish the usage data. During this step, the appropriate rates for

each service defined on our Rate Plan will be applied to the usage information, and the resulting charge is

then loaded into the database.

The best part is that you only have to go through this process once. When the workbook is completed, you

just need to add it to the scheduler and it will run hourly or daily at the scheduled time. Then it’s just a

matter of creating new reports, or customizing existing ones, in order to visualize and analyze the data to

get meaningful insights from your cloud usage.

Next Steps

For more information about Cloud Cruiser, please contact info@cloudcruiser.com.

For detailed instructions on creating Cloud Cruiser workbooks and reports, please reference Cloud Cruiser’s

online documentation (valid login required).

To learn how to integrate with the Azure Rate Card API, refer to the online tutorial Cloud Cruiser and Azure

Public Rate Card API Integration