provision software-defined storage infrastructure at …...overview of the design and development of...
Post on 14-Jul-2020
0 Views
Preview:
TRANSCRIPT
© 2020 Cisco and/or its affiliates. All rights reserved. Page 1 of 20
Provision Software-Defined Storage Infrastructure at
Cloud Scale with Terraform Provider for Cisco Intersight
White paper
Public
© 2020 Cisco and/or its affiliates. All rights reserved. Page 2 of 20
Contents
Executive summary 3
Introduction 3
Solution deployment 6
Conclusion 19
For more information 19
About the Author 20
© 2020 Cisco and/or its affiliates. All rights reserved. Page 3 of 20
This document describes how to use the Terraform provider for the Cisco Intersight™ to
achieve automated, zero-touch infrastructure deployment for cloud-scale software-defined
storage on the Cisco Unified Computing System™ (Cisco UCS®). This guide provides an
overview of the design and development of a Terraform configuration for provisioning
infrastructure and installing the operating system for a generic software-defined storage
solution using the Terraform provider for the Cisco Intersight platform.
Executive summary
This document describes how to use the Terraform provider for Cisco Intersight™ software with the Cisco
Unified Computing System™ (Cisco UCS®) platform to automate infrastructure provisioning and operating system
deployment for cloud-scale storage. This guide provides the framework for deploying any software-defined
storage solution on three Cisco UCS C240 M5 Rack Server nodes with two Cisco UCS C220 M5 Rack Servers
as controller and load-balancer nodes. Cisco UCS provides computing, network, and storage components as a
unified platform for software-defined storage solutions. Cisco Intersight software provides a systems
management platform that delivers intuitive computing through cloud-powered intelligence. The Cisco
Intersight platform provides infrastructure management for Cisco UCS and Cisco HyperFlex™ systems. It
supports the use of the Terraform provider to develop infrastructure as code, facilitating zero-touch
deployment.
Introduction
This section provides an overview of the Terraform provider, Cisco Intersight and Software-defined storage
solution with Cisco Intersight.
Cisco Intersight platform
The Cisco Intersight platform provides benefits for the entire infrastructure lifecycle. Cisco UCS Manager, Cisco
UCS Director, and Cisco Integrated Management Controller (IMC) focus on day-0 and day-1 activities related to
deployment and configuration. The Cisco Intersight platform augments this focus with benefits for day 2 and
beyond as well. In addition to monitoring and alerting, Cisco Intersight provides integration with the Cisco®
Technical Assistance Center (TAC), predictive analytics, and resource optimization to address ongoing
operations and systems upgrades.
Figure 1 shows a high-level view of multiple management stages with the Cisco Intersight platform.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 4 of 20
Figure 1.
Cisco Intersight infrastructure lifecycle
HashiCorp Terraform
Terraform is software for building complex, version-controlled, and collaborative infrastructure with high
productivity. The infrastructure Terraform can manage includes low-level components such as computing
instances, storage, and networking, as well as high-level components such as Domain Name System (DNS)
entries and software-as-a-service (SaaS) features.
The set of files used to describe infrastructure in Terraform is known as a Terraform configuration. The
configuration is written using HashiCorp Configuration Language (HCL), a simple human-readable configuration
language, to define a desired topology of infrastructure resources.
Why Terraform provider for the Cisco Intersight?
The Terraform provider allows organizations to develop Cisco Intersight resources as self-service infrastructure
using code rather than manual provisioning.
This approach provides several benefits:
● You can more quickly and easily scale Cisco Intersight resources. You can provision infrastructure in
minutes, with little effort, using the automated workflows, performing the same tasks that used to
take days.
● The operating model of Terraform is well suited for the Cisco Intersight platform, because it
accommodates the shift from static to dynamic infrastructure provisioning. For example, if a resource
is deleted in the Terraform configuration, it will be reflected in the Cisco Intersight platform when the
new configuration is applied.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 5 of 20
● Terraform maintains a state file, which is a record of the currently provisioned resources. State files
provide a version history of Cisco Intersight resources, enabling a detailed audit trail of changes.
● The provider enables idempotency, producing the same result and state with repeated API calls.
Software-defined storage solution with Cisco UCS and Cisco Intersight
The Cisco Intersight platform provides a comprehensive architecture for deploying and managing software-
defined storage. Figure 2 shows the hardware design for a generic software-defined storage solution.
Figure 2.
Solution hardware design
The design includes the following components:
● The Cisco Intersight platform is deployed as a SaaS solution.
● Controller and load-balancer nodes are deployed on Cisco UCS C220 M5 servers.
● Storage nodes are deployed on Cisco UCS C240 M5 servers.
● Cisco UCS C240 and C220 servers are connected to Cisco Nexus® 93240YC-FX2 Switches with 25-
Gbps line speed.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 6 of 20
Solution deployment
The deployment of the whole solution consists of several main steps:
● Create an account on the Cisco Intersight platform and claim all storage nodes.
● Create a Terraform configuration environment for the Cisco Intersight platform.
● Apply Terraform configurations for the following:
◦ Update firmware on all storage nodes.
◦ Create server policies and profiles.
◦ Deploy server profiles for all nodes.
◦ Install the operating system on all nodes.
● Deploy software-defined storage software.
Creating an account on the Cisco Intersight platform and claiming nodes
You create an account on the Cisco Intersight platform by claiming a Cisco IMC device. Before you claim the
device, you should perform a preliminary check of the Cisco Intersight device connector in the IMC. The current
device connector requirements are described in
https://www.cisco.com/c/en/us/td/docs/unified_computing/Intersight/b_Cisco_Intersight_Appliance_Getting_
Started_Guide/b_Cisco_Intersight_Appliance_Install_and_Upgrade_Guide_chapter_010.pdf and summarized in
Table 1.
Table 1. Device connector requirements
Component Minimum software version Supported device connector version
Releases that include supported device connectors
Cisco UCS Manager Release 3.2(I) Release 1.0.9-2290 Release 4.0(2a) or later
Cisco IMC Supervisor For M5 servers: Release 3.1(3a)
For M4 servers: Release 3.0(4)
Release 1.0.9-335 Release 4.0(2c) or later
Cisco HyperFlex Connect and Cisco HyperFlex HX Data Platform
Release 2.6 Release 1.0.9-1335 Release 3.5(2a) or later
To create an account on the Cisco Intersight platform, follow these steps:
1. Log in to https://www.intersight.com/.
2. Sign in with your Cisco ID (Figure 3).
© 2020 Cisco and/or its affiliates. All rights reserved. Page 7 of 20
Figure 3.
Cisco Intersight main screen
3. Read the offer description and accept it.
4. Copy the device ID and claim code in the Cisco Intersight Account Creation wizard from the Cisco
IMC.
5. Enter a name for the account and click Create.
The device claim process can take a few minutes. If required, the device connector will automatically be upgraded as part of the process. Account creation and basic configuration are now finished.
6. On the Cisco Intersight dashboard, choose Devices > Claim a New Device (Figure 4).
Figure 4.
Claiming a new device
7. Copy the device ID and claim code for the second Cisco UCS C240 M5L server and click Claim.
8. Repeat the same process for the third Cisco UCS C240 M5L server and for the Cisco UCS C220 M5
servers.
9. After you claim all the devices, you should see the devices listed under the server tab at the left.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 8 of 20
Create a Terraform configuration environment for the Cisco Intersight platform
Next you create the Terraform configuration environment.
Install Terraform
On the deployment host, make sure that Terraform is installed. The Terraform binary is available for major
distributions.
1. Download the Terraform zip file from https://www.terraform.io/downloads.html according to the
operating system of your deployment host.
2. Extract the zip file and move it to a directory of your choice.
3. Add the path of the directory to the PATH variable of the system.
For more information about how to install Terraform, see https://learn.hashicorp.com/terraform/getting-
started/install.html.
Clone the repository
Use the following command to clone the repository on your deployment host. This repository contains code to
deploy the whole solution.
# git clone https://github.com/ucs-compute-solutions/terraform-intersight-sds.git
After the repository has been cloned, you should see five directories: firmware_update, create_infra,
provision_infra, os_deployment, and unbind_profiles. Separate directories are created for logical separation
and easy understanding of workflows.
If required, you can merge all the resources into a single file in your deployment with fewer modifications
Copy the Terraform provider binary file
The Terraform binary file for the Cisco Intersight platform is built for Microsoft Windows, Darwin (Mac OS), and
Linux 64-bit architecture systems. Choose the appropriate binary file according to the operating system in your
deployment host.
Follow these steps:
4. Go to https://github.com/cisco-intersight/terraform-provider-intersight.
5. Navigate to the Releases tab. Download the terraform-provider-intersight.zip file for the latest
release.
6. Extract the files and navigate to the terraform-provider-intersight binary file for the operating system
installed on your deployment host.
7. Copy the terraform-provider-intersight binary file to all the directories of the downloaded repository.
Generate API keys
To use the Cisco Intersight provider, you need an API key, a secret key, and the Cisco Intersight endpoint URL.
To generate the keys, follow these steps:
8. Log in to https://www.intersight.com/.
9. On the Settings screen, click the Settings menu.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 9 of 20
10. On the General page, choose API > API Keys > Generate API Key.
11. On the Generate New API Key screen, enter the purpose for the API key and click Generate. The API
key ID and RSA private key are displayed.
12. Copy the API key.
13. Save the private key information in a .pem file. Save it in a location in the downloaded repository.
Define the Cisco Intersight provider
Now define the Cisco Intersight provider.
Navigate to the create_infra directory in the cloned repository and open main.tf file. Enter the API key you
copied. Also provide the secret key file. The endpoint changes if you are using Cisco Intersight appliance.
provider "intersight" {
secretkeyfile = "Secret key file generated from previous step "
endpoint = "https://intersight.com"
}
For simplicity in this document, multiple directories are defined for each workflow (to create infrastructure,
apply server profiles, upgrade firmware, etc.). Copy the same main.tf file in all the directories. You can also
combine all the Terraform configurations in one directory.
Configure variables
You need to define some of the basic inputs required for various workflows to provision the infrastructure. You
will define all these as variables in a file named variables.tf.
Each configuration takes a value in the variables.tf file as shown here:
variable "variable_name " {
default = <value of the variable>
}
You need to keep several image files, such as the operating system image file, Cisco Server Configuration
Utility (SCU), and Cisco Host Update Utility (HUU), in a remote server for firmware upgrades and operating
system installation. Download the files from software.cisco.com and keep them in a local Network File System
(NFS), HTTP, or Common Internet File System (CIFS) share that is accessible to the Cisco IMC for the servers.
Then you can edit the variables for the remote server and images.
In variables.tf, define all the VLAN IDs that you need for the management, client, cluster, and replication
networks.
Now you need the managed object ID (MOID) for the organization and claimed storage nodes. To get these
values, follow these steps:
14. Log in to https://www.intersight.com/.
15. From the Help screen, click Get More Help from Cisco Intersight.
16. Navigate to API Documentation.
17. Click API Reference (Figure 5).
© 2020 Cisco and/or its affiliates. All rights reserved. Page 10 of 20
Figure 5.
API Reference
18. Search for compute/PhysicalSummary.
19. Click GET: Read a 'compute.PhysicalSummary' resource (Figure 6).
Figure 6.
Supported methods for compute/PhysicalSummary resource
© 2020 Cisco and/or its affiliates. All rights reserved. Page 11 of 20
20. In the representational state transfer (REST) client, click Send to get a response from the
compute/PhysicalSummary API (Figure 7).
Figure 7.
REST Client screen
21. The response is an array of physical servers claimed with all the information about the claimed
servers. The first entry is the MOID for the server. Copy the MOIDs for all the claimed servers and
the organization MOID.
22. You have all the basic input required to apply the Terraform configurations. Copy the same
varables.tf file in all the directories.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 12 of 20
Understanding the Cisco Intersight provider and Terraform configuration
The resource objects names in the configuration are in this format:
intersight_<model_name_in_snake_case>
For example, intersight_ntp_policy is the resource object for the Network Time Protocol (NTP) policy, and
intersight_storage_disk_group_policy is the resource object for the storage disk group policy. The following is
an example of a complete resource definition for a server profile and the NTP policy attached to the server
profile:
resource "intersight_server_profile" "storage-node1" {
name = "storage-node1"
organization {
object_type = "organization.Organization"
moid = var.org_moid
}
assigned_server {
moid = var.storage-node1
object_type = "compute.RackUnit"
}
}
resource "intersight_ntp_policy" "sds-ntp-policy" {
name = "sds-ntp-policy"
enabled = true
ntp_servers = [
"ntp.esl.cisco.com",
"171.68.38.65",
"173.38.201.115"
]
organization {
object_type = "organization.Organization"
moid = var.org_moid
}
profiles {
moid = intersight_server_profile.storage-node1.moid
object_type = "server.Profile"
}
}
© 2020 Cisco and/or its affiliates. All rights reserved. Page 13 of 20
Each resource is assigned a name, which can later be used for tracking and referencing. This name will not be
reflected anywhere in the Cisco Intersight platform. It is only for reference among the .tf files. The NTP policy is
attached to the server profile created earlier. This is accomplished by referencing the storage-node1 profile
in profiles.moid. A resource can point to or reference another resource using the
format <resource>.<resource_name>.<property_name>.
View the logs
TF_LOG is a terraform variable that is used for viewing different categories of logs. By default, this variable is
left empty. To view logs for Terraform operations, this variable must be set to DEBUG.
In Mac OS and Linux, you accomplish this with export TF_LOG=debug. In Windows PowerShell, use
$env:TF_LOG=“DEBUG”.
View documentation
Documentation about provider resources and configuration options can be found at https://github.com/cisco-
intersight/terraform-provider-intersight/tree/master/website/docs.
Implement the Terraform configuration
You need to run terraform init whenever you start with new Terraform code.
The terraform init command will scan the code, identify the provider you are using, and download the
appropriate provider. In the case here, because you copied the provider manually, Terraform use the Cisco
Intersight provider.
You next run the terraform plan command.
The plan command allows you to see what Terraform will do to the resources before any changes are actually
made. This command provides a good way to check the potential results before you make any changes to the
infrastructure.
In the output, the symbols show you the following:
● Resources with a plus sign (+) will be created.
● Resources with a minus sign (-) will be deleted.
● Resources with a tilde (~) will be modified in place.
Finally, run terraform apply to deploy the configuration. Observe that the apply command shows the
same output as the plan command. To actually proceed with the plan, you need to type yes to confirm.
Applying Terraform configurations for workflows
Next apply the Terraform configurations to the various workflows.
Upgrade firmware
You should upgrade the firmware for the Cisco UCS C240 M5 servers to a suggested release. On the Cisco
Intersight platform, you upgrade firmware using a noninteractive Cisco HUU to upgrade the BIOS, Cisco IMC,
PCI adapters, RAID controllers, and other firmware to compatible versions.
You can upgrade firmware either through a network share or using utility storage. In this example, the firmware
is upgraded through a network share using the NFS protocol. The HUU image file is mounted directly in the
Cisco IMC. Therefore, it requires uninterrupted connectivity between the remote file server and Cisco IMC.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 14 of 20
To apply the Terraform configuration for firmware update, follow these steps:
23. Go to the firmware_update directory.
24. Append all the servers that require firmware updates to firmware_update.tf.
25. If running the configuration for the first time, run terraform init.
26. Run terraform apply to initiate the firmware updates.
Create server policies
You need multiple server policies for any generic software-defined storage solution. Table 2 lists the policies
required and the corresponding resource objects in the Terraform provider for the Cisco Intersight platform.
Table 2. Terraform provider policies and resource objects
Policy Terraform rResource oObject Comments
Adapter configuration
intersight_adapter_config_policy Specify the ID of the PCI slot ID in which the Cisco virtual interface card (VIC) adapter is placed.
Ethernet adapter intersight_vnic_eth_adapter_policy
Ethernet network intersight_vnic_eth_network_policy Create these resources based on the number of networks required for the software-defined storage solution. In the example here, four networks are being created.
Ethernet quality of service
intersight_vnic_eth_qos_policy
LAN connectivity intersight_vnic_lan_connectivity_policy
intersight_vnic_eth_if
The first resource creates LAN connectivity policy. The second resource creates multiple virtual interfaces.
NTP intersight_ntp_policy
Disk group intersight_storage_disk_group_policy The purpose of this resource is to create RAID from 2 solid-state disks (SSDs) present in the server. This resource will be used for OS installation.
Storage intersight_storage_storage_policy This resource will include disk group policy and also set policies for virtual drives. This resource will also set storage disks to JBOD mode.
Boot order intersight_boot_precision_policy
To apply the Terraform configuration to create server policies, follow these steps:
27. Go to the create_infra directory.
28. Update the server_profiles.tf file with the number of server profiles required.
29. If you are running the configuration for the first time, run terraform init.
30. Run terraform apply to initiate the firmware updates.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 15 of 20
31. After the updates are applied, verify that all the server profiles are created in the Cisco Intersight
portal by navigating to Policies > Server Policies (Figure 8).
Figure 8.
Policies created by Terraform configuration
Associate server profiles
After all the required polices have been created, you need to associate the server profiles with each physical
server. To associate the profiles with the servers, follow these steps:
32. Go to the provision_infra directory.
33. Update the server_profiles.tf file with the resource intersight_server_profile for each server profile.
34. For each server profile, define the server with which you want to associate it in the assigned_server
attribute.
35. If you are running the configuration for the first time, run terraform init.
36. Run terraform apply to initiate the firmware updates.
37. The process for applying the server profiles takes a few minutes. You can see that association is in
progress from the Requests screen (Figure 9).
© 2020 Cisco and/or its affiliates. All rights reserved. Page 16 of 20
Figure 9.
Server profile association in progress
38. After the server profiles have been applied, verify server profile association in the Cisco Intersight
portal by navigating to Profiles > Server Profiles (Figure 10).
Figure 10.
Server profile associated with physical servers
Make changes to resources
If any of the resources needs updating, you first need to unbind the server profile, then make the changes, and
then bind the server profiles back again.
Follow these steps to make any changes:
39. Go to the unbind_profiles directory.
40. Update unbind_profiles.tf file with the server profiles that you want to unbind.
41. Run terraform apply.
42. Make any necessary changes in either policies or profiles.
43. Go to provision_infra directory and run terraform apply.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 17 of 20
Install the operating system
The operating system used for this document is Red Hat Enterprise Linux (RHEL) 7.6. However, you can choose
a different operating system based on the Cisco UCS Hardware and Software Compatibility Matrix and support
from your storage software vendor.
To install the OS, follow these steps:
44. Go to the os_deployment directory.
45. In repo_setup.tf file, define all the OS images in the resource
intersight_softwarerepository_operating_system_file. Use a separate resource for each OS type.
46. In the same file, configure the resource for the Cisco SCU in the
intersight_firmware_server_configuration_utility_distributable resource.
47. In os_install.tf file, define the resource intersight_os_install for each node.
48. Run terraform apply to install the OS.
Complying with the Cisco Hardware Compatibility List
The Cisco Intersight platform evaluates the compatibility of your Cisco UCS and Cisco HyperFlex systems to
verify that the hardware and software have been tested and validated by Cisco or Cisco partners. The Cisco
Intersight platform reports validation issues after checking the compatibility of the server hardware (server
model, CPU, and server firmware version), server software (current OS vendor and OS version), and adapter
compliance (adapter model, driver protocol, and driver version for the firmware).
You can manually determine the recommended hardware and firmware versions for your server configuration
with the Cisco UCS Hardware and Software Compatibility tool at
https://ucshcltool.cloudapps.cisco.com/public/.
The Cisco Intersight platform provides am open-source tool called the OS Discovery Tool to collect the OS and
driver information need to evaluate Cisco Hardware Compatibility List (HCL) compliance in Linux operating
system versions (Figure 11).
Figure 11.
OS Discovery Tool
© 2020 Cisco and/or its affiliates. All rights reserved. Page 18 of 20
Follow these steps to evaluate compliance with the HCL:
49. Set up any Linux virtual or physical machine that has access to the on-premise network. Install
Python 2.7 on this control node.
50. On this control node, clone the Python software development kit (SDK) repository for the Cisco
Intersight platform:
git clone https://github.com/CiscoUcs/intersight-python.git
51. Install the Python SDK:
cd intersight-python
python setup.py install
52. In the intersight-python directory, go to os-discovery-tool.
53. Edit the discovery_config_linux.json file to include intersight_api_key and path
intersight_secret_file.
54. Run the following script:
./get_linux_inv_to_intersight.py --log-inventory --
configfile=discovery_config_linux.json
55. The script collects information about the operating system and device drivers on all the nodes and
routes this information to the Cisco Intersight platform to help evaluate compliance with the HCL.
After this process is complete, you should be able to see the compliance status. In the Servers list,
choose one of the servers and navigate to the HCL tab. You should see the HCL status (Figure 12).
Figure 12.
Hardware and software compatibility status
56. For all the components, you can also see recommended versions and download the drivers by
clicking Get Recommended Drivers (Figure 13).
© 2020 Cisco and/or its affiliates. All rights reserved. Page 19 of 20
Figure 13.
HCL recommendations
Deploy software-defined storage software
After the server profiles are attached and operating system is installed on all the nodes, you can deploy your
software-defined storage solution. Refer to the software-defined storage vendor’s installation guides to deploy
the solution.
Typically, you need to install two components: controllers and load balancer nodes and claiming storage nodes.
You can customize your Terraform resources, such as the number of networks and LAN connectivity policy,
based on the requirements of your storage software vendor.
Conclusion
The Cisco Intersight SaaS platform transforms the way that customers deploy and manage Cisco UCS and
Cisco HyperFlex systems. The Cisco Intersight platform supports the Terraform provider, which offers an
excellent way to easily build, scale, and manage the lifecycle of any scale-out storage software solution with
Cisco UCS servers connected to a switched environment. The Cisco Intersight platform offers resources for
updating firmware, configuring profiles, associating profiles with physical servers, managing profiles, and
installing the operating system.
For more information
For additional information, see the following resources:
● Terraform provider for the Cisco Intersight platform on GitHub:
https://github.com/cisco-intersight/terraform-provider-intersight
● GitHub repository for the solution:
https://github.com/ucs-compute-solutions/terraform-intersight-sds
© 2020 Cisco and/or its affiliates. All rights reserved. Page 20 of 20
● Cisco Intersight online help: https://intersight.com/help/home
● Cisco Intersight data sheet: https://www.cisco.com/c/en/us/products/collateral/servers-unified-
computing/intersight/datasheet-c78-739433.html#FlexibleDeploymentOptions
About the Author
Paniraja Koppa, Cisco Systems, Inc.
Paniraja Koppa is a Technical Marketing Engineer for UCS Solutions. He has more than 13 years of experience
with a primary focus on data center technologies such as Cisco UCS, Storage, Operating systems, Automation,
Virtualization and Cloud. In his current role at Cisco Systems, he works on best practices, optimization,
automation and performance tuning of software defined storage on Cisco UCS platforms. Prior to this, he has
led QA efforts for 4 new virtual adapter card’s firmware and software features for Cisco UCS. He also worked
as customer support engineer and advocate in the Data Center Virtualization space.
Acknowledgements
For their support and contribution, the authors would like to thank:
● Vikrant Balyan, Cisco Systems, Inc.
● Aanisha Mishra, Cisco Systems, Inc.
● Chris O'Brien, Cisco Systems, Inc.
● Oliver Walsdorf, Cisco Systems, Inc.
● Jawwad Memon, Cisco Systems, Inc.
● David Soper, Cisco Systems, Inc.
Printed in USA 220104.1 05/20
top related