ibm operations analytics predictive insights...operations analytics predictive insights azure...

Operations Analytics Predictive Insights

IBM Operations Analytics Predictive Insights Mediation Pack for Azure

Version 1.1.0

Operations Analytics Predictive Insights Azure Mediation Pack

2

Edition notice

This edition applies to IBM Operations Analytics Predictive Insights version 1.3.3 and to all subsequent releases.

Please refer to the IBM developerWorks terms of use for more information. Also reference the IBM Operations

Analytics Development Community (https://developer.ibm.com/itoa/) for additional information on Predictive

Insights.

http://www.ibm.com/developerworks/community/terms/

https://developer.ibm.com/itoa/


3

Contents Overview ................................................................................................................................................ 3

Prerequisite Information ........................................................................................................................ 4

Azure Performance Metrics .................................................................................................................... 5

Mediation Packs Contents ................................................................................................................... 7

Configure Data Modeling ................................................................................................................... 10

Create Topic ...................................................................................................................................... 13

Deploy Model .................................................................................................................................... 14

Start Training ..................................................................................................................................... 14

Review Training Process .................................................................................................................... 15

Review Anomalies ............................................................................................................................. 15

Support ................................................................................................................................................. 18

Notices .................................................................................................................................................. 19

Overview IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a system

through KPI metric data streams. This includes data from various sources including metrics gathered by


4

industry performance and network monitoring solutions. A mediation pack helps users quickly integrate

the KPI data streams by providing a default model to integrate a performance manager. IBM Operations

Analytics Predictive Insights Mediation Pack for Azure provides a pre-built model for data ingestion.

Predictive Insights creates a performance model that it uses to detect or forecast behavior outside the

modeled range, including generating an alarm when an anomalous behavior occurs. Doing so enables

organizations to pro-actively avoid system and solution outages before users are impacted. The

Mediation Pack for Azure provides the capability to ingest performance data and metrics from Azure

into IBM Operations Analytics Predictive Insights to perform analysis. This document will cover the steps

to setup the Mediation Pack and adjust the data stream for deployment in a customer environment.

Azure Mediation Pack Architecture (CSV)

Mediation Pack development considers all possible integration options to determine the optimal

configuration. Options are reviewed based on the source systems ability to present data using multiple

methods. The selected methods are derived based on many factors including but not limited to

flexibility, scalability, and availability of KPI’s (Key Performance Indicators). Additional considerations are

based on the source systems majority configuration and maturity of such methods at the time of

development. Requests to enhance mediation packs utilizing additional methods supported by the

source system can be reviewed with our professional services team.

Prerequisite Information


5

Item Description Name Mediation Pack for Azure

Java IBM Java 8 / oracle 1.8

OS Linux

Predictive Insights supported Versions V1.3.3, V1.3.5 and 1.3.6.

Predictive Insights Information IBM Operations Analytics (demo, trial, resources)

https://www.ibm.com/us-en/marketplace/it-operations-analytics

IBM Operations Analytics Community (docs, videos, mediation packs) https://developer.ibm.com/itoa/

IBM Operations Analytics Predictive Insights documentation

http://www.ibm.com/support/knowledgecenter/SSJQQ3

Filename Azure_MedPack_v1.1.0.zip

Data source Every 5 minutes, the run.sh script in this pack extracts the

csv files which later collected by Predictive Analytics.

• Azure needs to be configured for historical

Additional Information collection for metrics and collecting performance metric data for every 5 min

Azure Performance Metrics

List of Azure resource types that supports metrics along with the metric namespaces given in the

<install_Dir>/Azure_MedPack_v1.1.0/azure_metrics directory.

Here all_azure_metrics_05_20_2019.txt contains list of all available resource types along with

corresponding metric names.recommended_azure_metrics_05_20_2019.txt contains the metrics that

are recommended for azure with respect to the Operations Analytics Predictive Insights.

This pack was tested with resource type Microsoft.Compute/virtualMachines and corresponding

metrics.

Installing and Configuring

Get Azure Auth information for configuring pack, using below steps.


6

(Ref: https://docs.microsoft.com/en-us/rest/api/azure/)

Azure Cloud Shell

Go to Azure Cloud Shell

Local with Azure CLI

Install Azure CLI 2.0

Login

az login

Set Active Subscription

az account set --subscription "your subscription name or id

Create Service Principal

az ad sp create-for-rbac -n "your service principal name"

Copy this output to a temp location, you will need the values for Pack configuration

Service Principal Password Reset

You can execute the following command if you ever need to reset your Service Principal password.

az ad sp reset-credentials --name "your service principal name"

IBM JAVA (1.8 and above)

Make Sure Java installed on the machine where the pack is going to install. Verify the java version using

below command. Should give the out put of JRE version running on the machine.

Java -version


7

Mediation Packs Contents The following are contained in the Mediation Pack: Azure_MedPack_v1.1.0.zip

•all_azure_metrics_05_20_2019.txt

•recommended_azure_metrics_05_20_2019.txt

azure_metrics

•run.sh

/bin

•Azure_MediationPack_UserGuide_v1.1.0.pdf

•Azure_MediationPack_README_v1.1.0.txt

/docs

•authBean.class

•AzureMain.class

•getAuthToken.class

•getMetricData.class

•ListResource.class

•ListResourceBean.class

•MainObject.class

•Name.class

•ResourceValueBean.class

•Timeseries.class

•TimeseriesData.class

•tsToSec8601.class

•Value.class

classes

•application.properties

conf

•run.sh

bin

•Azure_MedPack.log

logs

•com.fasterxml.jackson.annotations.jar

•com.fasterxml.jackson.core.jar

•com.fasterxml.jackson.databind.jar

lib

etc

•azure.pamodel

/pamodel

•virtualMachines_2019_08_21_20_50_00.csv

/sample

license


8

/bin/run.sh - for run the mediation pack to collect the performance metrics.

/class - Directory contains the binary files needed by the Mediation Pack.

/conf - Directory contains the config file for the pack.

/lib - Directory contains the jar files

/logs/Azure_MedPack.log - Directory contains the logfile of the Mediation Pack.

/docs - Users Guide for Mediation Pack of Azure for deploying and configuring the Mediation Pack.

Azure_MediationPack_UserGuide_v1.1.0.pdf, Azure_MediationPack_README_v1.1.0

This is a deployment procedural guide to deploy the Azure mediation pack into the Predictive

Insights environment.

/license - Text license files for governing the use of the Mediation pack.

/pamodel - Core files used in the Mediation Pack representing the Predictive Insights model consisting of

all attributes and metric definitions.

azure.pamodel

The azure.pamodel file is a Predictive Insights model file which consists of all attributes and metric

definitions. This file consists of all metric group definitions. Import this file into mediation tool.

/sample

Contains a sample CSV file representing the captured data streams that can be used to try the Mediation

Pack output in the Operations Analytics Predictive Insights product. There will be one csv file for each

resource type.


9

Mediation Pack property file configuration

tokenUrl: Which is equal to https://login.microsoftonline.com/{tenantId}/oauth2/token

{tenantId} id replace with the value with above genrated tenantId.

(look into the step "Create Service Principal" in the output token is the tenantId)

grant_type: Which is equal to "client_credentials"

client_id: Which is equal to the appId genrated in the step "Create Service Principal"

client_secret: Which is equal to the password genrated in the step "Create Service Principal"

resource: Which is equal to "https://management.azure.com"

subscription_id: subscription id by running the following command

az account show --query id

csv_dir: Which is the directory where the generated csv files will save.

resource_types: List of resources types for which the metric information needed with (,) separated.

After resource_types, Give each resource type with required metrics as given in the default

application.properties file.

List of All azure metrics and recommended metrics were given in the

<install_Dir>/Azure_MedPack_1.1.0/azure_metrics directory. Copy the line for which resource type

required and past into the property file after resource_type property .user may not need to provide all

the metrics of resource type but the one requires to monitor. Like the list of recommended metrics that

are provided above.

timespan: The length of the interval to query. 5 minutes is the default in the application.properties file.

Latency: This is the difference between the current time to the timespan. It is better to gather the

metrics for few minutes past rather than last 5 minutes from the current time in order to be confident

that all the data for that interval is available in Azure

Configuring Mediation Pack in Cron

Provide execute permission to run.sh using below command

chmod 775 <install_Dir>Azure_MedPack_1.1.0/bin/run.sh

To collect the data for every 5 minutes interval. run.sh script should be run in cron.

Below is the sample cron expression to run for every five minutes.

*/5 * * * * <install_Dir>Azure_MedPack_1.1.0/bin/run.sh

https://login.microsoftonline.com/%7btenantId%7d/oauth2/token


10

Configure Data Modeling

Follow the below procedure to configure the azure model

1. Copy the azure.pamodel file to the Predictive Insights server where the mediation tool

installed.

2. Launch the mediation tool using $PI_HOME/bin/mediationtool.sh

3. Go to File and click on import button

Note: The model is pre-defined with some selected metrics. Make sure the mediation model is updated with all metrics

and/or metric groups that are present in pack_filetered_metrics.py file before deployment. Refer ‘Configuration and

Administration Guide’ of IBM Operations Analytics Predictive Insights for model configuration (chapter 2).


11

4. Select the Predictive Insights Model in the menu and click on Next

5. Select the azure.pamodel file and click on ok


12

6. Click on finish to complete the import process

7. Enter the azure Predictive Insights Data source connection details.

For more information refer to the following url:

https://www.ibm.com/support/knowledgecenter/SSJQQ3_1.3.6/com.ibm.scapi.doc/admin_gui

de/t_tasp_admin_createfilesystemdatasource.html

10. Change to the model design tab you will able to see all metric groups

11. Select each metric group in turn and preview the data by clicking the preview button near the

top-right.

https://www.ibm.com/support/knowledgecenter/SSJQQ3_1.3.6/com.ibm.scapi.doc/admin_guide/t_tasp_admin_createfilesystemdatasource.html

https://www.ibm.com/support/knowledgecenter/SSJQQ3_1.3.6/com.ibm.scapi.doc/admin_guide/t_tasp_admin_createfilesystemdatasource.html


13

Create Topic

Follow below procedure to create topic in Predictive Insights.

1. Login into the Predictive Insights server as scadmin

2. Change to $PI_HOME/bin

3. Create the topic using ./admin.sh create_topic azure

4. ./admin.sh set -t=azure system.aggregation.interval 5

For more information on properties for the admin.sh refer to the following:

http://www.ibm.com/support/knowledgecenter/SSJQQ3_1.3.6/com.ibm.scapi.doc/adm

in_guide/r_tsaa_adminguide_cli_show_instance_properties.html

Refer to the following for the set parameter:


in_guide/r_tsaa_adminguide_cli_set_property.html

Refer to the following on the properties that can be set:


in_guide/c_tsaa_adminguide_config_properties.html

http://www.ibm.com/support/knowledgecenter/SSJQQ3_1.3.6/com.ibm.scapi.doc/adm%20in_guide/r_tsaa_adminguide_cli_show_instance_properties.html

http://www.ibm.com/support/knowledgecenter/SSJQQ3_1.3.6/com.ibm.scapi.doc/adm%20in_guide/r_tsaa_adminguide_cli_show_instance_properties.html


14

Deploy Model

1. Click on the model(s) to be deployed and multi-select if multiple models are to be deployed to

the same topic. Then Right click on the selection and select Deploy Model option.

2. Provide the Predictive Insights hostname, port number and database name

3. Provide the password for scadmin and click on ok

4. Click on Yes to deploy the model

Start Training 1. Login into the Predictive Insights server as scadmin

2. Change to $PI_HOME/bin

3. Run ./start.sh -t=azure to start the topic

4. Start the training using below command. -s start time is based on your datasource collection.

-l is the time it takes for an interval to have fully arrived from the datasource.

For example ./admin.sh run_extractor_instance -t=azure -s=20170917-1610 -l=60


15

Review Training Process

Once training has started you will see events in the Active event list

1. Login in to https://hostname:16311/ibm/console

2. User name: ncoadmin

3. Enter ncoadmin password

4. You can see the list events showing status about training process in Active event list

Review Anomalies 1. Login in to https://hostname:16311/ibm/console

2. User name: ncoadmin

3. Enter ncoadmin password

4. Navigate to the Detected Anomalies Page

https://hostname:16311/ibm/console


16

5. Review the Anomalies

6. Right click on an anomaly and select Service Diagnosis tool


17

7. Review the affected metrics and behavior of the metrics for the period


18

Support

To get assistance from the IBM support team, you may open a Problem Management Record (PMR).

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM Operations

Analytics Advanced Insights indicating the "Mediation Pack for Solution Name". Describe the problem

encountered providing the following information that must be attached to the PMR:

Product Name: IBM Operations Analytics Predictive Insights or IBM Operations Analytics Advanced

Insights

Product Version: 1.3.3 or higher.

Mediation Pack: Mediation Pack for Azure

Mediation Pack Version: 1.1.0

Include the following:

1. Sample extracted CSV files, which are being ingested into Predictive Insights.

Attach the output of the command $PI_HOME/bin/collect.sh -t=<topic>

2. Data source connectivity status screenshot

3. Predictive Insights log files ($PI_HOME/log/topicname/) when the problem occurred while

using the mediation pack.

4. A screen shot of the error message that is generated while using the mediation pack.

5. A specific use case scenario(s) where problem occurred while using the mediation pack.

6. Azure_MedPack.log file after running the pack. Which will be in <Install_Dir>/logs/ directory.


19

Notices

© Copyright IBM Corporation 2019

IBM United States of America

Produced in the United States of America

US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for

information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to

state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does

not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation

of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does

not give you any license to these patents. You can send license inquiries, in writing, to:

IBM Director of Licensing IBM Corporation

North Castle Drive Armonk, NY 10504-1785 U.S.A.

For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual Property Department in your country or send

inquiries, in writing, to:

Intellectual Property Licensing

Legal and Intellectual Property Law

IBM Japan, Ltd.

1623-14, Shimotsuruma, Yamato-shi

Kanagawa 242-8502 Japan

The following paragraph does not apply to the United Kingdom or any other country where such provisions are

inconsistent with local law:

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these

changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the

program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of

those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between

independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged,

should contact:


20

IBM Corporation 2Z4A/101

11400 Burnet Road Austin, TX 78758 U.S.A.

Such information may be available, subject to appropriate terms and conditions, including in some cases payment of a fee.

The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer

Agreement, IBM International Program License Agreement or any equivalent agreement between us.

Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating

environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that

these measurements will be the same on generally available systems. Furthermore, some measurement may have been estimated through

extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly

available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related

to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and

objectives only.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the

examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names

and addresses used by an actual business enterprise is entirely coincidental.

If you are viewing this information in softcopy form, the photographs and color illustrations might not be displayed.

Copyright License

This information contains sample application programs in source language, which illustrate programming techniques on various operating

platforms. You may copy modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing,

using, marketing or distributing application programs conforming to the application programming interface for the operating platform for

which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot

guarantee or imply reliability, serviceability, or function of these programs. The sample programs are provided "AS IS", without warranty of any

kind. IBM shall not be liable for any damages arising out of your use of the sample programs.

Privacy Policy Considerations

IBM Software products, including software as service solutions, (“Software Offerings”) may use cookies or other technologies to collect product

usage information, to help improve the end user experience, to tailor interactions with the end user, or for other purposes. In many cases, no

personally identifiable information is collected by the Software Offerings. Some of our Software Offerings can help enable you to collect

personally identifiable information. If this Software Offering uses cookies to collect personally identifiable information, specific information

about this offering’s use of cookies is set forth below.

Depending on the configuration that is deployed in the application server, this Software Offering may use session cookies for session

management. These cookies can optionally be enabled or disabled, but disabling the cookies will also disable the functionality that they enable.

See the application server documentation for additional information.

Optionally, you can also use functionality, such as typeahead, to store data locally on the browser. The data that is stored locally could contain

personally identifiable information: Data validation.

If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable

information from end users by using cookies and other technologies, you should seek your own legal advice about any laws that are applicable

to such data collection, including any requirements for notice and consent.

For more information about the use of various technologies, including cookies, for these purposes, see IBM’s Privacy Policy

athttp://www.ibm.com/privacy and IBM's Online Privacy Statement at http://www.ibm.com/privacy/details in the section entitled “Cookies,

https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ibm.com%2Fprivacy&data=02%7C01%7CTVamsi%40vicomnet.com%7Ccb85d633315b4c809d3c08d732c80f44%7Ce2eac50279f7450193442bc870f9fcde%7C0%7C0%7C637033707134382946&sdata=hdJUIvN89nDn%2FgdGmpyQtnn8BzIiHwDQM%2BmDVkOhhyE%3D&reserved=0

https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ibm.com%2Fprivacy%2Fdetails&data=02%7C01%7CTVamsi%40vicomnet.com%7Ccb85d633315b4c809d3c08d732c80f44%7Ce2eac50279f7450193442bc870f9fcde%7C0%7C0%7C637033707134392969&sdata=bjpe23zaeDKdjXhwUaBoAjcB%2F7HOjI7s7T8JKFNHiMA%3D&reserved=0


21

Web Beacons and Other Technologies” and the "IBM Software Products and Software-as-a-Service Privacy Statement"

at http://www.ibm.com/software/info/product-privacy.

Trademarks

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many

jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies.

A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml.

Adobe, Acrobat, PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in

the United States, other countries, or both.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Linux is a trademark of

Linus Torvalds in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or

both.

UNIX is a registered trademark of The Open Group in the United States and other countries. Other product and service names might be

trademarks of IBM or other companies.

https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ibm.com%2Fsoftware%2Finfo%2Fproduct-privacy&data=02%7C01%7CTVamsi%40vicomnet.com%7Ccb85d633315b4c809d3c08d732c80f44%7Ce2eac50279f7450193442bc870f9fcde%7C0%7C0%7C637033707134402969&sdata=OvJQJenQrIhMKLbSSWWw2tCjKzRViVzonOw2x9ktSqQ%3D&reserved=0

https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fibm.com&data=02%7C01%7CTVamsi%40vicomnet.com%7Ccb85d633315b4c809d3c08d732c80f44%7Ce2eac50279f7450193442bc870f9fcde%7C0%7C0%7C637033707134402969&sdata=jWJ7U1WsIJ%2F9CQ33wv%2FTu2%2BBKeR2kiB6vomXETl9v8E%3D&reserved=0

https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ibm.com%2Flegal%2Fcopytrade.shtml&data=02%7C01%7CTVamsi%40vicomnet.com%7Ccb85d633315b4c809d3c08d732c80f44%7Ce2eac50279f7450193442bc870f9fcde%7C0%7C0%7C637033707134412977&sdata=RRI1zwZcROkvWJ0Dcs7mZz6oaHwIlHY95J4hrl1pkGQ%3D&reserved=0

ibm operations analytics predictive insights...operations analytics predictive insights azure...

Documents