vmware validated design reference architecture...

VMware Validated Design™ for Micro-Segmentation

Reference Architecture Guide

VMware Validated Design for Micro-Segmentation 3.0

This document supports the version of each product listed and supports all subsequent versions until the document is replaced by a new edition. To check for more recent editions of this document, see http://www.vmware.com/support/pubs.

EN-002236-00

http://www.vmware.com/support/pubs

VMware Validated Design for Micro-Segmentation Reference Architecture Guide

© 2016 VMware, Inc. All rights reserved.

Page 2 of 130

The VMware Web site also provides the latest product updates.

If you have comments about this documentation, submit your feedback to: [email protected]

© 2016 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. This product is covered by one or more patents listed at http://www.vmware.com/download/patents.html.

VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.

VMware, Inc. 3401 Hillview Avenue Palo Alto, CA 94304 www.vmware.com

mailto:[email protected]://www.vmware.com/download/patents.html



Page 3 of 130

Contents

1. Purpose and Intended Audience ...................................................... 9

2. Architecture Overview .................................................................... 10

2.1 Physical Infrastructure Architecture ............................................................................. 11

2.1.1 Pod Architecture .................................................................................................................... 11

2.1.2 Physical Network Architecture ............................................................................................... 13

2.1.3 Availability Zones and Regions ............................................................................................. 19

2.2 Virtual Infrastructure Architecture ................................................................................ 21

2.2.1 Infrastructure Architecture ..................................................................................................... 21

2.2.2 Virtual Infrastructure Overview .............................................................................................. 21

2.2.3 Network Virtualization Architecture ........................................................................................ 23

2.2.4 Logging Architecture ............................................................................................................. 28

2.2.5 Operations Management Architecture ................................................................................... 30

3. Detailed Design .............................................................................. 33

3.1 Physical Infrastructure Design ..................................................................................... 33

3.1.1 Physical Design Fundamentals ............................................................................................. 34

3.1.2 Physical Networking Design .................................................................................................. 40

3.1.3 Physical Storage Design ....................................................................................................... 49

3.2 Virtual Infrastructure Design ........................................................................................ 57

3.2.1 Virtual Infrastructure Design Overview .................................................................................. 58

3.2.2 ESXi Design .......................................................................................................................... 61

3.2.3 vCenter Server Design .......................................................................................................... 63

3.2.4 Virtualization Network Design................................................................................................ 77

3.2.5 NSX Design ........................................................................................................................... 91

3.2.6 Shared Storage Design ....................................................................................................... 111

3.3 Operations Infrastructure Design ............................................................................... 121

3.3.1 vRealize Log Insight Design ................................................................................................ 121



Page 4 of 130

List of Tables

Table 4. vRealize Operations Manager Logical Node Architecture ...................................................... 32

Table 5. Regions ................................................................................................................................... 34

Table 6. Availability Zones and Regions Design Decisions .................................................................. 35

Table 7. Required Number of Racks ..................................................................................................... 36

Table 8. POD and Racks Design Decisions ......................................................................................... 37

Table 9. ESXi Host Design Decisions ................................................................................................... 39

Table 10. Host Memory Design Decision .............................................................................................. 40

Table 11. Jumbo Frames Design Decisions ......................................................................................... 44

Table 12. VLAN Sample IP Ranges ...................................................................................................... 46

Table 13. Physical Network Design Decisions...................................................................................... 48

Table 14. Additional Network Design Decisions ................................................................................... 49

Table 15. Virtual SAN Physical Storage Design Decision .................................................................... 50

Table 16. Virtual SAN Mode Design Decision ...................................................................................... 50

Table 17. Hybrid and All-Flash Virtual SAN Endurance Classes ......................................................... 52

Table 18. SSD Endurance Class Design Decisions ............................................................................. 52

Table 19. SSD Performance Classes ................................................................................................... 53

Table 20. SSD Performance Class Selection ....................................................................................... 53

Table 21. SSD Performance Class Design Decisions .......................................................................... 54

Table 22. Virtual SAN HDD Environmental Characteristics .................................................................. 54

Table 23. HDD Characteristic Selection ............................................................................................... 55

Table 24. HDD Selection Design Decisions.......................................................................................... 55

Table 25. NFS Usage Design Decisions ............................................................................................... 56

Table 26. NFS Hardware Design Decision ........................................................................................... 57

Table 27. Volume Assignment Design Decisions ................................................................................. 57

Table 28. ESXi Boot Disk Design Decision ........................................................................................... 62

Table 29. ESXi User Access Design Decisions .................................................................................... 63

Table 30. Other ESXi Host Design Decisions ....................................................................................... 63

Table 31. vCenter Server Design Decision ........................................................................................... 64

Table 32. vCenter Server Platform Design Decisions .......................................................................... 65

Table 33. Platform Service Controller Design Decisions ...................................................................... 65

Table 34. Methods for Protecting vCenter Server System and the vCenter Server Appliance ............ 66

Table 35. vCenter Server Systems Protection Design Decisions ......................................................... 67

Table 36. Logical Specification for Management vCenter Server Appliance ........................................ 67

Table 37. Logical Specification for Compute vCenter Server Appliance .............................................. 67

Table 38. vCenter Appliance Sizing Design Decisions ......................................................................... 68

Table 39. vCenter Database Design Decisions .................................................................................... 69

Table 40. vSphere HA Design Decisions .............................................................................................. 70



Page 5 of 130

Table 41. vSphere Cluster Workload Design Decisions ....................................................................... 71

Table 42. Management Cluster Design Decisions ................................................................................ 71

Table 43. Management Cluster Attributes ............................................................................................ 72

Table 44. Edge Cluster Design Decisions ............................................................................................ 73

Table 45. Shared Edge and Compute Cluster Attributes ...................................................................... 74

Table 46. Compute Cluster Design Decisions ...................................................................................... 75

Table 47. Monitor Virtual Machines Design Decisions ......................................................................... 75

Table 48. vSphere Distributed Resource Scheduling Design Decisions .............................................. 76

Table 49. VMware Enhanced vMotion Compatibility Design Decisions ............................................... 76

Table 50. vCenter Server TLS Certificate Design Decisions ................................................................ 76

Table 51. Virtual Switch Design Decisions ........................................................................................... 79

Table 52. Virtual Switch for the Management Cluster .......................................................................... 79

Table 53. vDS-Mgmt Port Group Configuration Settings ...................................................................... 80

Table 54. Management Virtual Switches by Physical/Virtual NIC......................................................... 80

Table 55. Management Virtual Switch Port Groups and VLANs .......................................................... 81

Table 56. Management VMkernel Adapter ........................................................................................... 81

Table 57. Virtual Switch for the shared Edge and Compute Cluster .................................................... 82

Table 58. vDS-Comp01 Port Group Configuration Settings ................................................................. 82

Table 59. Shared Edge and Compute Cluster Virtual Switches by Physical/Virtual NIC ..................... 83

Table 60. Edge Cluster Virtual Switch Port Groups and VLANs........................................................... 83

Table 61. Shared Edge and Compute Cluster VMkernel Adapter ........................................................ 84

Table 62. Virtual Switches for Compute Cluster Hosts ......................................................................... 84

Table 63. vDS-Comp02 Port Group Configuration Settings ................................................................. 84

Table 64. Compute Cluster Virtual Switches by Physical/Virtual NIC .................................................. 85

Table 65. Compute Cluster Virtual Switch Port Groups and VLANs .................................................... 86

Table 66. Compute Cluster VMkernel Adapter ..................................................................................... 86

Table 67. NIC Teaming and Policy ....................................................................................................... 87

Table 68. NIC Teaming Design Decision .............................................................................................. 87

Table 69. Network I/O Control Design Decision ................................................................................... 88

Table 70. VXLAN Design Decisions ..................................................................................................... 90

Table 71. NSX for vSphere Design Decision ........................................................................................ 91

Table 72. Consumption Method Design Decisions ............................................................................... 93

Table 73. NSX Controller Design Decision ........................................................................................... 94

Table 74. NSX for vSphere Physical Network Requirements ............................................................... 95

Table 75. Resource Specification of NSX Components ....................................................................... 96

Table 76. NSX Edge Service Gateway Sizing Design Decision ........................................................... 97

Table 77. vSphere Compute Cluster Split Design Decisions ................................................................ 99

Table 78. VTEP Teaming and Failover Configuration Design Decision ............................................. 101

Table 79. Logical Switch Control Plane Mode Decision ..................................................................... 102



Page 6 of 130

Table 80. Transport Zones Design Decisions ..................................................................................... 103

Table 81. Routing Model Design Decision .......................................................................................... 103

Table 82. Transit Network Design Decision ........................................................................................ 105

Table 83. Tenant Firewall Design Decision ........................................................................................ 105

Table 84. Load Balancer Features of NSX Edge Services Gateway ................................................. 106

Table 85. NSX for vSphere Load Balancer Design Decision .............................................................. 107

Table 86.Virtual to Physical Interface Type Design Decision ............................................................. 107

Table 87. Inter-Site Connectivity Design Decisions ............................................................................ 108

Table 88. Isolated Management Applications Design Decisions ........................................................ 108

Table 90. Application Virtual Network Configuration .......................................................................... 111

Table 91. Network Shared Storage Supported by ESXi Hosts ........................................................... 112

Table 92. vSphere Features Supported by Storage Type .................................................................. 112

Table 93. Storage Type Design Decisions .......................................................................................... 114

Table 94. VAAI Design Decisions ....................................................................................................... 115

Table 95. Virtual Machine Storage Policy Design Decisions .............................................................. 116

Table 96. Storage I/O Control Design Decisions ................................................................................ 116

Table 97. Resource Management Capabilities Available for Datastores ........................................... 117

Table 112. NFS Version Design Decision ........................................................................................... 118

Table 113. NFS Export Sizing ............................................................................................................. 119

Table 114. NFS Export Design Decisions ........................................................................................... 120

Table 115. NFS Datastore Design Decision ....................................................................................... 120

Table 179. Cluster Node Configuration Design Decision ................................................................... 122

Table 180. Compute Resources for a vRealize Log Insight Medium-Size Node ................................ 123

Table 181. Compute Resources for the vRealize Log Insight Nodes Design Decision ...................... 123

Table 182. vRealize Log Insight Isolated Network Design Decisions ................................................. 124

Table 183. IP Subnets in the Application Isolated Networks .............................................................. 124

Table 184. IP Subnets Design Decision ............................................................................................. 125

Table 185. DNS Names of the vRealize Log Insight Nodes ............................................................... 125

Table 186. DNS Names Design Decision ........................................................................................... 125

Table 187. Virtual Disk Configuration in the vRealize Log Insight Virtual Appliance .......................... 126

Table 188. Retention Period Design Decision .................................................................................... 126

Table 189. Log Archive Policy Design Decision ................................................................................. 127

Table 190. SMTP Alert Notification Design Decision .......................................................................... 127

Table 192. Custom Role-Based User Management Design Decision ................................................ 128

Table 193. Custom Certificates Design Decision ............................................................................... 128

Table 194. Direct Log Communication to vRealize Log Insight Design Decisions ............................. 128

Table 195. Time Synchronization Design Decision ............................................................................ 129

Table 196. Syslog Protocol Design Decision ...................................................................................... 129

Table 197. Protocol for Event Forwarding across Regions Design Decision ..................................... 130



Page 7 of 130

List of Figures

Figure 1. Overview of SDDC Architecture ............................................................................................ 10

Figure 2. Pods in the SDDC .................................................................................................................. 12

Figure 3. Leaf-and-Spine Physical Network Design ............................................................................. 13

Figure 4. High-level Physical and Logical Representation of a Leaf Node ........................................... 14

Figure 5. Pod Network Design .............................................................................................................. 16

Figure 6. Oversubscription in the Leaf Layer ........................................................................................ 17

Figure 7. Compensation for a Link Failure ............................................................................................ 18

Figure 8. Quality of Service (Differentiated Services) Trust Point ........................................................ 19

Figure 9. Availability Zones and Regions .............................................................................................. 20

Figure 10. Virtual Infrastructure Layer in the SDDC ............................................................................. 21

Figure 11. SDDC Logical Design .......................................................................................................... 22

Figure 12. NSX for vSphere Architecture .............................................................................................. 24

Figure 13. NSX for vSphere Universal Distributed Logical Router ....................................................... 27

Figure 19. Cluster Architecture of vRealize Log Insight ........................................................................ 29

Figure 20. vRealize Operations Manager Architecture ......................................................................... 31

Figure 21. Physical Infrastructure Design ............................................................................................. 33

Figure 22. Physical Layer within the SDDC .......................................................................................... 34

Figure 23. SDDC Pod Architecture ....................................................................................................... 35

Figure 24. Leaf-and-Spine Architecture ................................................................................................ 41

Figure 25. Example of a Small-Scale Leaf-and-Spine Architecture ..................................................... 42

Figure 26. Leaf-and-Spine and Network Virtualization ......................................................................... 43

Figure 27. Leaf Switch to Server Connection within Compute Racks .................................................. 44

Figure 28. Leaf Switch to Server Connection within Management/Shared Compute and Edge Rack . 45

Figure 29. Sample VLANs and Subnets within a Pod .......................................................................... 46

Figure 30. Oversubscription in the Leaf Switches................................................................................. 48

Figure 31. Virtual Infrastructure Layer Business Continuity in the SDDC ............................................ 58

Figure 32. SDDC Logical Design .......................................................................................................... 59

Figure 33. vSphere Data Protection Logical Design ............................................................................. 60

Figure 34. Disaster Recovery Logical Design ....................................................................................... 61

Figure 35. vCenter Server and Platform Services Controller Deployment Model ................................ 66

Figure 36. vSphere Logical Cluster Layout ........................................................................................... 69

Figure 37. Network Switch Design for Management Hosts .................................................................. 80

Figure 38. Network Switch Design for shared Edge and Compute Hosts ............................................ 83

Figure 39. Network Switch Design for Compute Hosts ......................................................................... 85

Figure 40. Architecture of NSX for vSphere .......................................................................................... 92

Figure 41. Conceptual Tenant Overview .............................................................................................. 98

Figure 42. Cluster Design for NSX for vSphere .................................................................................. 100



Page 8 of 130

Figure 43. Logical Switch Control Plane in Hybrid Mode .................................................................... 102

Figure 44. Virtual Application Network Components and Design ....................................................... 109

Figure 45. Detailed Example for vRealize Automation Networking .................................................... 110

Figure 46. Logical Storage Design ...................................................................................................... 113

Figure 49. NFS Storage Exports ......................................................................................................... 119

Figure 58. Operations Infrastructure Conceptual Design ................................................................... 121

Figure 59. Logical Design of vRealize Log Insight .............................................................................. 121

Figure 60. Networking Design for the vRealize Log Insight Deployment ........................................... 124



Page 9 of 130

1. Purpose and Intended Audience

VMware Validated Design for Micro-Segmentation Reference Architecture Guide contains a

validated model of the VMware Validated Design for Micro-Segmentation use case and provides

a detailed design of each management component of the stack.

The Architecture Overview discusses the building blocks and the main principles of each layer.

The Detailed Design provides the available design options according to the design objective, and

a set of design decisions to justify selecting the path for building each component.

The VMware Validated Design for Micro-Segmentation Reference Architecture Guide is

compliant and validated with certain product versions. See the VMware Validated Design for

Micro-Segmentation Planning and Preparation Guide for more information about supported

product versions.

The VMware Validated Design for Micro-Segmentation Reference Architecture Guide is intended for cloud architects, infrastructure administrators and cloud administrators who are familiar with and want to use VMware software to deploy in a short time and manage an SDDC that meets the requirements for capacity, scalability, backup and restore, and extensibility for disaster recovery support.

Note Design decisions in this document are based on design decisions in the VMware

Validated Design Reference Architecture Guide, but some decision have been removed or

changed. As a result, the decisions are not always numbered consecutively.

Note Illustrations in this document show an architecture that includes VMware Virtual SAN.

The Validated Design for Micro-Segmentation does not include Virtual SAN, but can be

expanded to use Virtual SAN.



Page 10 of 130

2. Architecture Overview

The VMware Validated™ Design for Micro-Segmentation enables an IT organization to automate the provisioning of common repeatable requests and to respond to business needs with more agility and predictability.

The architecture of the full VMware Validated Design is based on a number of layers and modules, which allows interchangeable components be part of the end solution or outcome such as the SDDC. If a particular component design does not fit the business or technical requirements for whatever reason, it should be able to be swapped out for another similar component. The VMware Validated Designs are one way of putting an architecture together. They are rigorously tested to ensure stability, scalability and compatibility. Ultimately however, the system is designed in such a way as to ensure the desired IT outcome will be achieved.

The VMware Validated Design for Micro-Segmentation does not include all layers of the illustration below, but can be expanded to include all layers.

Figure 1. Overview of SDDC Architecture

Physical Layer

The lowest layer of the solution is the Physical Layer, sometimes referred to as the 'core', which consists of three main components, Compute, Network and Storage. Inside the compute component sit the x86 based servers that run the management, edge and tenant compute workloads. There is some guidance around the capabilities required to run this architecture, however no recommendations on the type or brand of hardware is given. All components must be supported on the VMware Hardware Compatibility guide.

Virtual Infrastructure Layer

Sitting on the Physical Layers infrastructure is the Virtual Infrastructure Layer. Within the Virtual Infrastructure Layer, access to the physical underlying infrastructure is controlled and allocated to the management and tenant workloads. The Virtual Infrastructure Layer consists primarily of the physical host's hypervisor and the control of these hypervisors. The management workloads consist of elements in the virtual management layer itself, along with elements in the Cloud Management Layer, Service Management, Business Continuity and Security areas.

Cloud Management Layer

The Cloud Management Layer is the "top" layer of the stack and is where the service consumption occurs. This layer is not part of the VMware Validated Design for Micro-Segmentation but is part of the software-defined data center. Typically, through a UI or API, this layer calls for resources and then



Page 11 of 130

orchestrates the actions of the lower layers to achieve the request. While the SDDC can stand on its own without any other ancillary services, for a complete SDDC experience other supporting components are needed. The Service Management, Business Continuity and Security areas complete the architecture by providing this support.

Service Management

When building any type of IT infrastructure, portfolio and operations management play key roles in continued day-to-day service delivery. The Service Management area of this architecture mainly focuses on operations management in particular monitoring, alerting and log management. Portfolio Management is not a focus of this SDDC design but may be added in future releases.

Business Continuity

To ensure a system that is enterprise ready, it must contain elements to support business continuity in the area of backup, restore and disaster recovery. This area ensures that when data loss occurs, the right elements are in place to ensure there is no permanent loss to the business. The design provides comprehensive guidance on how to operate backups, restore and also the run books of how to fail components over in the event of a disaster.

Security

All systems need to inherently be secure by design to reduce risk and increase compliance while still providing a governance structure. The security area outlines what is needed to ensure the entire SDDC is resilient to both internal and external threats.

2.1 Physical Infrastructure Architecture

2.1.1 Pod Architecture

The VMware Validated Design for SDDC uses a small set of common building blocks called pods. Pods can include different combinations of servers, storage equipment, and network equipment, and can be set up with varying levels of hardware redundancy and varying quality of components. Pods are connected to a network core that distributes data between them. The pod is not defined by any hard physical properties, as it is a standard unit of connected elements within the SDDC network fabric.

2.1.1.1. Pod

A pod is a logical boundary of functionality for the SDDC platform. While each pod usually spans one rack, it is possible to aggregate multiple pods into a single rack in smaller setups. For both small and large setups, homogeneity and easy replication are important.

Different pods of the same type can provide different characteristics for varying requirements. For example, one compute pod could use full hardware redundancy for each component (power supply through memory chips) for increased availability. At the same time, another compute pod in the same setup could use low-cost hardware without any hardware redundancy. With these variations, the architecture can cater to the different workload requirements in the SDDC.

One of the guiding principles for such deployments is that VLANs are not spanned beyond a single pod by the network virtualization layer. Although this VLAN restriction appears to be a simple requirement, it has widespread impact on how a physical switching infrastructure can be built and on how it scales.

2.1.1.2. Pod Types

The SDDC differentiates between the following types of pods:

Management pod

Shared Edge and Compute pod

Compute pod

Storage pod



Page 12 of 130

Figure 2. Pods in the SDDC

2.1.1.3. Management Pod

The management pod runs the virtual machines that manage the SDDC. These virtual machines host vCenter Server, NSX Manager, NSX Controller, vRealize Operations Management, vRealize Log Insight, vRealize Automation, and other shared management components. Different types of management pods can support different SLAs. Because the management pod hosts critical infrastructure, you should consider implementing a basic level of hardware redundancy for this pod.

Management pod components must not have tenant-specific addressing.

2.1.1.4. Shared Edge and Compute Pod

The shared edge and compute pod runs the required NSX services to enable north-south routing between the SDDC and the external network, and east-west routing inside the SDDC. This shared pod also hosts the SDDC tenant virtual machines (sometimes referred to as workloads or payloads). As the SDDC grows, additional compute-only pods can be added to support a mix of different types of workloads for different types of Service Level Agreements (SLAs).

2.1.1.5. Compute Pod

Compute pods host the SDDC tenant virtual machines (sometimes referred to as workloads or payloads). An SDDC can mix different types of compute pods and provide separate compute pools for different types of SLAs.

2.1.1.6. Storage Pod

Storage pods provide network-accessible storage using NFS or iSCSI. Different types of storage pods can provide different levels of SLA, ranging from just a bunch of disks (JBODs) using IDE drives with minimal to no redundancy, to fully redundant enterprise-class storage arrays. For bandwidth-intense IP-based storage, the bandwidth of these pods can scale dynamically.

This design does not consider Fibre Channel or Fibre Channel over Ethernet (FCoE) based storage technology. Instead, this design focuses on technologies that can use the central Layer 3 network fabric for primary connectivity.

2.1.1.7. Pod to Rack Mapping

Pods are not mapped one-to-one to 19" data center racks. While a pod is an atomic unit of a repeatable building block, a rack is merely a unit of size. Because pods can have different sizes, how pods are mapped to 19" data center racks depends on the use case.



Page 13 of 130

One Pod in One Rack. One pod can occupy exactly one rack. This is typically the case for compute pods.

Multiple Pods in One Rack. Two or more pods can occupy a single rack, for example, one management pod and one shared edge and compute pod can be deployed to a single rack.

Single Pod Across Multiple Racks. A single pod can stretch across multiple adjacent racks. For example, a storage pod with filer heads and disk shelves can span more than one rack.

2.1.2 Physical Network Architecture

The physical network architecture is tightly coupled with the pod-and-core architecture, and uses a Layer 3 leaf-and-spine network instead of the more traditional 3-tier data center design.

2.1.2.1. Leaf-and-Spine Network Architecture

The design uses leaf switches and spine switches.

A leaf switch is typically located inside a rack and provides network access to the servers inside that rack, it is also referred to as a Top of Rack (ToR) switch.

A spine switch is in the spine layer and provides connectivity between racks. Links between spine switches are typically not required. If a link failure occurs between a spine switch and a leaf switch, the routing protocol ensures that no traffic for the affected rack is sent to the spine switch that has lost connectivity to that rack.

Figure 3. Leaf-and-Spine Physical Network Design

Ports that face the servers inside a rack should have a minimal configuration, shown in the following high-level physical and logical representation of the leaf node.

Note Each leaf node has identical VLAN configuration with unique /24 subnets assigned to each VLAN.



Page 14 of 130

Figure 4. High-level Physical and Logical Representation of a Leaf Node

2.1.2.2. Network Transport

You can implement the physical layer switch fabric for an SDDC by offering Layer 2 transport services or Layer 3 transport services to all components. For a scalable and vendor-neutral data center network, use Layer 3 transport.

2.1.2.3. Benefits and Drawbacks for Layer 2 Transport

In a design that uses Layer 2 transport, leaf switches and spine switches form a switched fabric, effectively acting like one large switch. Using modern data center switching fabric products such as Cisco FabricPath, you can build highly scalable Layer 2 multipath networks without the Spanning Tree Protocol (STP). Such networks are particularly suitable for large virtualization deployments, private clouds, and high-performance computing (HPC) environments.

Using Layer 2 routing has benefits and drawbacks.

The benefit of this approach is more design freedom. You can span VLANs, which is useful for vSphere vMotion or vSphere Fault Tolerance (FT).

The drawback is that the size of such a deployment is limited because the fabric elements have to share a limited number of VLANs. In addition, you have to rely on a specialized data center switching fabric product from a single vendor because these products are not designed for interoperability between vendors.

2.1.2.4. Benefits and Drawbacks for Layer 3 Transport

A design with Layer 3 transport requires these considerations.

Layer 2 connectivity is limited to within the data center rack, up to the leaf switch.

The leaf switch terminates each VLAN and provides default gateway functionality, that is, it has a switch virtual interface (SVI) for each VLAN.

Uplinks from the leaf switch to the spine layer are routed point-to-point links. VLAN trunking on the uplinks is not allowed.



Page 15 of 130

A dynamic routing protocol—for example OSPF, ISIS, or iBGP—connects the leaf switches and spine switches. Each leaf switch in the rack advertises a small set of prefixes, typically one per VLAN or subnet. In turn, the leaf switch calculates equal cost paths to the prefixes it received from other leaf switches

Using Layer 3 routing has benefits and drawbacks.

The benefit is that you can chose from a wide array of Layer 3 capable switch products for the physical switching fabric. You can mix switches from different vendors due to general interoperability between implementation of OSPF, ISIS or iBGP. This approach is usually more cost effective because it uses only basic functionality of the physical switches.

The drawbacks are some design restrictions because VLANs are restricted to a single rack. This affects vSphere vMotion, vSphere Fault Tolerance, and storage networks.

2.1.2.5. Infrastructure Network Architecture

One of the key goals of network virtualization is to provide a virtual-to-physical network abstraction. For this, the physical fabric must provide a robust IP transport with the following characteristics.

Simplicity

Scalability

High bandwidth

Fault-tolerant transport

Support for different levels of quality of service (QoS)

Simplicity

Configuration of the switches inside a data center must be simple. General or global configuration such as AAA, SNMP, syslog, NTP, and others should be replicated line by line, independent of the position of the switches. A central management capability to configure all switches at once is an alternative.

Scalability

Scalability factors include but are not limited to the following.

Number of racks supported in a fabric.

Amount of bandwidth between any two racks in a data center.

Number of paths that a leaf switch can select from when communicating with another rack.

The total number of ports available across all spine switches and the oversubscription that is acceptable determine the number of racks supported in a fabric. Different racks might host different types of infrastructure, which results in different bandwidth requirements.

Racks with storage systems might attract or source more traffic than other racks.

Compute racks, such as racks hosting hypervisors with workloads or virtual machines, might have different bandwidth requirements than shared edge and compute racks, which provide connectivity to the outside world.

Link speed and the number of links vary to satisfy different bandwidth demands. You can vary them for each rack without sacrificing other aspects of the leaf-and-spine architecture.



Page 16 of 130

Figure 5. Pod Network Design

The number of links to the spine switches dictates how many paths for traffic from this rack to another rack are available. Because the number of hops between any two racks is consistent, the architecture can utilize equal-cost multipathing (ECMP). Assuming traffic sourced by the servers carries a TCP or UDP header, traffic spray can occur on a per-flow basis.

High Bandwidth

In leaf-and-spine topologies, oversubscription typically occurs at the leaf switch.

Oversubscription is equal to the total amount of bandwidth available to all servers connected to a leaf switch divided by the aggregate amount of uplink bandwidth.

oversubscription = total bandwidth / aggregate uplink bandwidth

For example, 20 servers with one 10 Gigabit Ethernet (10 GbE) port each create up to 200 Gbps of bandwidth. In an environment with eight 10 GbE uplinks to the spine—a total of 80 Gbps—a 2.5:1 oversubscription results shown in the Oversubscription in the Leaf Layer illustration.

2.5 (oversubscription) = 200 (total) / 80 (total uplink)



Page 17 of 130

Figure 6. Oversubscription in the Leaf Layer

You can make more or less bandwidth available to a rack by provisioning more or fewer uplinks. That means you can change the available bandwidth on a per-rack basis.

Note The number of uplinks from a leaf switch to each spine switch must be the same to avoid hotspots.

For example, if a leaf switch has two uplinks to spine switch A and only one uplink to spine switches B, C and D, more traffic is sent to the leaf switch via spine switch A, which might create a hotspot.

Fault Tolerance

The larger the environment, the more switches make up the overall fabric and the greater the possibility for one component of the data center switching fabric to fail. A resilient fabric can sustain individual link or box failures without widespread impact.



Page 18 of 130

Figure 7. Compensation for a Link Failure

For example, if one of the spine switches fails, traffic between racks continues to be routed across the remaining spine switches in a Layer 3 fabric. The routing protocol ensures that only available paths are chosen. Installing more than two spine switches reduces the impact of a spine switch failure.

Multipathing-capable fabrics handle box or link failures, reducing the need for manual network maintenance and operations. If a software upgrade of a fabric switch becomes necessary, the administrator can take the node out of service gracefully by changing routing protocol metrics, which will quickly drain network traffic from that switch, freeing the switch for maintenance.

Depending on the width of the spine, that is, how many switches are in the aggregation or spine layer, the additional load that the remaining switches must carry is not as significant as if there were only two switches in the aggregation layer. For example, in an environment with four spine switches, a failure of a single spine switch only reduces the available capacity by 25%.

Quality of Service Differentiation

Virtualized environments must carry different types of traffic, including tenant, storage and management traffic, across the switching infrastructure. Each traffic type has different characteristics and makes different demands on the physical switching infrastructure.

Management traffic, although typically low in volume, is critical for controlling physical and virtual network state.

IP storage traffic is typically high in volume and generally stays within a data center.

For virtualized environments, the hypervisor sets the QoS values for the different traffic types. The physical switching infrastructure has to trust the values set by the hypervisor. No reclassification is necessary at the server-facing port of a leaf switch. If there is a congestion point in the physical switching infrastructure, the QoS values determine how the physical network sequences, prioritizes, or potentially drops traffic.



Page 19 of 130

Figure 8. Quality of Service (Differentiated Services) Trust Point

Two types of QoS configuration are supported in the physical switching infrastructure:

Layer 2 QoS, also called class of service

Layer 3 QoS, also called DSCP marking.

A vSphere Distributed Switch supports both class of service and DSCP marking. Users can mark the traffic based on the traffic type or packet classification. When the virtual machines are connected to the VXLAN-based logical switches or networks, the QoS values from the internal packet headers are copied to the VXLAN-encapsulated header. This enables the external physical network to prioritize the traffic based on the tags on the external header.

2.1.2.6. Server Interfaces (NICs)

If the server has more than one server interface (NIC) of the same speed, use two as uplinks with VLANs trunked to the interfaces.

The vSphere Distributed Switch supports many different NIC Teaming options. Load-based NIC teaming supports optimal use of available bandwidth and supports redundancy in case of a link failure. Use two 10 GbE connections per server along with a pair of leaf switches. 801.Q trunks are used for carrying a small number of VLANs; for example, management, storage, VXLAN, vSphere Replication, and VMware vSphere vMotion traffic.

2.1.3 Availability Zones and Regions

In an SDDC, availability zones are collections of infrastructure components. Regions support disaster recovery solutions and allow you to place workloads closer to your customers. Typically, multiple availability zones form a single region.



Page 20 of 130

This VMware Validated Design uses two regions, but uses only one availability zone in each region. The following diagram shows how the design could be expanded to include multiple availability zones.

Figure 9. Availability Zones and Regions

2.1.3.1. Availability Zones

Each availability zone is isolated from other availability zones to stop the propagation of failure or outage across zone boundaries. Together, multiple availability zones provide continuous availability through redundancy, helping to avoid outages and improve SLAs. An outage that is caused by external factors (power, cooling, physical integrity) affects only one zone, those factors most likely don't lead to an outage in other zones except in the case of major disasters.

Each availability zone runs on its own physically distinct, independent infrastructure, and is engineered to be highly reliable. Each zone should have independent power, cooling, network and security. Common points of failures within a physical data center, like generators and cooling equipment, should not be shared across availability zones. Additionally, these zones should be physically separate; so that even uncommon disasters affect only a single availability zone. Availability zones are usually either two distinct data centers within metro distance (latency in the single digit range) or two safety/fire sectors (aka data halls) within the same large scale data center.

Multiple availability zones (usually two) belong to a single region. The physical distance between availability zones can be up to approximately 50 kilometer or 30 miles, therefore offering low single-digit latency and large bandwidth - via dark fiber - between the zones. This architecture allows the SDDC equipment across the availability zone to operate in an active/active manner as a single virtual data center or region.

You can operate workloads across multiple availability zones within the same region as if they were part of a single virtual data center. This supports an architecture with very high availability that is suitable for mission critical applications. When the distance between two locations of equipment becomes too large, these locations can no longer function as two availability zones within the same region and need to be treated as separate regions.

2.1.3.2. Regions

Multiple regions support placing workloads closer to your customers, for example, by operating one region on the US east coast and one region on the US west coast, or operating a region in Europe and another region in the US. Regions are helpful in many ways.

Regions can support disaster recovery solutions: One region can be the primary site and another region can be the recovery site.

You can use multiple regions to address data privacy laws and restrictions in certain countries by keeping tenant data within a region in the same country.

The distance between regions can be rather large. This design uses two regions, one region is assumed to be in San Francisco (SFO), the other region is assumed to be in Los Angeles (LAX).



Page 21 of 130

2.2 Virtual Infrastructure Architecture

2.2.1 Infrastructure Architecture

The virtual infrastructure is the foundation of an operational SDDC. Within the virtual infrastructure layer, access to the physical underlying infrastructure is controlled and allocated to the management and tenant workloads. The virtual infrastructure layer consists primarily of the physical hosts' hypervisors and the control of these hypervisors. The management workloads consist of elements in the virtual management layer itself, along with elements in the cloud management layer and in the service management, business continuity, and security areas.

Figure 10. Virtual Infrastructure Layer in the SDDC

2.2.2 Virtual Infrastructure Overview

The SDDC virtual infrastructure consists of two regions. Each region includes a management pod and a shared edge and compute pod.



Page 22 of 130

Figure 11. SDDC Logical Design

2.2.2.1. Management Pod

Management pods run the virtual machines that manage the SDDC. These virtual machines host vCenter Server, NSX Manager, NSX Controller, vRealize Operations, vRealize Log Insight, vRealize Automation, Site Recovery Manager and other shared management components. All management, monitoring, and infrastructure services are provisioned to a vCenter Server High Availability cluster which provides high availability for these critical services. Permissions on the management cluster limit access to only administrators. This limitation protects the virtual machines that are running the management, monitoring, and infrastructure services.

2.2.2.2. Shared Edge and Compute Pod

The shared edge and compute pod runs the required NSX services to enable north-south routing between the SDDC and the external network and east-west routing inside the SDDC. This pod also hosts the SDDC tenant virtual machines (sometimes referred to as workloads or payloads). As the SDDC grows additional compute-only pods can be added to support a mix of different types of workloads for different types of SLAs.



Page 23 of 130

2.2.3 Network Virtualization Architecture

2.2.3.1. NSX for vSphere Components

VMware NSX for vSphere, the network virtualization platform, is a key solution in the SDDC architecture. The NSX for vSphere platform consists of several components that are relevant to the network virtualization design.

NSX for vSphere Platform

NSX for vSphere creates a network virtualization layer. All virtual networks are created on top of this layer, which is an abstraction between the physical and virtual networks. Several components are required to create this network virtualization layer.

vCenter Server

NSX Manager

NSX Controller

NSX Virtual Switch

NSX for vSphere API

These components are separated into different planes to create communications boundaries and provide isolation of workload data from system control messages.

Data plane. Workload data is contained wholly within the data plane. NSX logical switches segregate unrelated workload data. The data is carried over designated transport networks in the physical network. The NSX Virtual Switch, distributed routing, and the distributed firewall are also implemented in the data plane.

Control plane. Network virtualization control messages are located in the control plane. Control plane communication should be carried on secure physical networks (VLANs) that are isolated from the transport networks that are used for the data plane. Control messages are used to set up networking attributes on NSX Virtual Switch instances, as well as to configure and manage disaster recovery and distributed firewall components on each ESXi host.

Management plane. The network virtualization orchestration happens in the management plane. In this layer, cloud management platforms such as VMware vRealize® Automation™ can request, consume, and destroy networking resources for virtual workloads. Communication is directed from the cloud management platform to vCenter Server to create and manage virtual machines, and to NSX Manager to consume networking resources.

The different planes are connected through the APIs, which include REST, VMware vSphere, and VMware VIX APIs. The API used depends on the component being controlled.

NSX Manager

NSX Manager provides the centralized management plane for the NSX for vSphere architecture and has a one-to-one mapping with vCenter Server for workloads. NSX Manager performs the following functions.

Provides a single point of configuration and the REST API entry points in a vSphere environment configured for NSX for vSphere.

Responsible for deploying NSX Controller clusters, NSX Edge distributed routers, and NSX Edge services gateways (as appliances in OVF format), guest introspection services, and so on.

Responsible for preparing ESXi hosts for NSX for vSphere by installing VXLAN, distributed routing, and firewall kernel modules, as well as the User World Agent (UWA).

Communicates with NSX Controller clusters through REST and with the ESXi hosts through the VMware vFabric® RabbitMQ message bus. This internal message bus is specific to NSX for vSphere and does not require setup of additional services.

Generates certificates for the NSX Controller nodes and ESXi hosts to secure control plane communications with mutual authentication.



Page 24 of 130

VMware NSX allows linking multiple vCenter and VMware NSX deployments, and manage them from a single NSX Manager that is designated as primary. Such a linked environment includes both an NSX Manager primary instance, and one or more secondary instances.

The primary NSX Manager instance is linked to the primary vCenter Server instance and allows the creation and management of universal logical switches, universal (distributed) logical routers and universal firewall rules.

Each secondary NSX Manager instance can manage networking services that are local to itself. Up to seven secondary NSX Manager instances can be associated with the primary NSX Manager in a linked environment. You can configure network services on all NSX Manager instances from one central location.

Note A linked environment still requires one vCenter Server instance for each NSX Manager instance.

To manage all NSX Manager instances from the primary NSX Manager in a Cross-vCenter VMware NSX deployment, the vCenter Server instances must be connected with Platform Services Controller nodes in Enhanced Linked Mode.

Figure 12. NSX for vSphere Architecture

Control Plane



Page 25 of 130

VMware NSX supports three different replication modes to provide multiple destination communication.

Multicast Mode. When multicast replication mode is selected for a logical switch, VMware NSX relies on the Layer 2 and Layer 3 multicast capability of the physical network to ensure VXLAN encapsulated multi-destination traffic is sent to all the VXLAN tunnel end points (VTEPs). The control plane uses multicast IP addresses on the physical network in this mode.

Unicast Mode. In unicast mode, the control plane is managed by the NSX Controller instances and all replication is done locally on the host. No multicast IP addresses or physical network configurations are required. This mode is well suited for smaller deployments.

Hybrid Mode. Hybrid mode is an optimized version of unicast mode, where local traffic replication for the subnet is offloaded to the physical network. This mode requires IGMP snooping on the first hop switch, and an IGMP querier must be available. Protocol-independent multicast (PIM) is not required.

NSX Controller

The NSX Controller cluster is the control plane component, and is responsible for managing the switching and routing modules in the hypervisors. An NSX Controller node performs the following functions.

Provides the control plane to distribute VXLAN and logical routing information to ESXi hosts.

Clusters nodes for scale-out and high availability.

Slices network information across nodes in a cluster for redundancy purposes.

Eliminates the need for multicast support from the physical network infrastructure.

Provides ARP-suppression of broadcast traffic in VXLAN networks.

NSX for vSphere control plane communication occurs over the management network. Network information from the ESXi hosts and the distributed logical router control VMs is reported to NSX Controller instances through the UWA. The NSX Controller command line supports retrieval of VXLAN and logical routing network state information.

Data Plane

The NSX data plane consists of the NSX vSwitch, which is based on the vSphere Distributed Switch (VDS) and includes additional components. These components include kernel modules, which run within the ESXi kernel and provide services such as the distributed logical router (DLR) and distributed firewall (DFW). The NSX kernel modules also enable Virtual Extensible LAN (VXLAN) capabilities.

The NSX vSwitch abstracts the physical network and provides access-level switching in the hypervisor. It is central to network virtualization because it enables logical networks that are independent of physical constructs such as a VLAN. The NSX vSwitch provides multiple benefits.

Three types of overlay networking capabilities:

o Creation of a flexible logical Layer 2 overlay over existing IP networks on existing physical infrastructure, without the need to rearchitect the data center networks.

o Support for east/west and north/south communication while maintaining isolation between tenants.

o Support for application workloads and virtual machines that operate as if they were connected to a physical Layer 2 network.

Support for VXLAN and centralized network configuration.

A comprehensive toolkit for traffic management, monitoring and troubleshooting within a virtual network which includes Port Mirroring, NetFlow/IPFIX, configuration backup and restore, network health check, and Quality of Service (QoS).

In addition to the NSX vSwitch, the data plane also includes gateway devices, which can provide Layer 2 bridging from the logical networking space (VXLAN) to the physical network (VLAN). The



Page 26 of 130

gateway device is typically an NSX Edge Gateway device. NSX Edge Gateway devices offer Layer 2, Layer 3, perimeter firewall, load-balancing, Virtual Private Network (VPN), and Dynamic Host Control Protocol (DHCP) services.

Consumption Plane

In the consumption plane, different users of NSX for vSphere can access and manage services in different ways:

NSX administrators can manage the NSX environment from the vSphere Web Client.

End-users can consume the network virtualization capabilities of NSX for vSphere through the Cloud Management Platform (CMP) when deploying applications.

2.2.3.2. Network Virtualization Services

Network virtualization services include logical switches, logical routers, logical firewalls, and other components of NSX for vSphere.

Logical Switches

NSX for vSphere logical switches create logically abstracted segments to which tenant virtual machines can connect. A single logical switch is mapped to a unique VXLAN segment ID and is distributed across the ESXi hypervisors within a transport zone. This allows line-rate switching in the hypervisor without creating constraints of VLAN sprawl or spanning tree issues.

Universal Distributed Logical Router

The NSX for vSphere Universal Distributed Logical Router is optimized for forwarding in the virtualized space (between VMs, on VXLAN- or VLAN-backed port groups). Features include:

High performance, low overhead first hop routing.

Scaling the number of hosts.

Support for up to 1,000 logical interfaces (LIFs) on each distributed logical router.

The Universal Distributed Logical Router is installed in the kernel of every ESXi host, as such it requires a VM to provide the control plane. The universal distributed logical router Control VM is the control plane component of the routing process, providing communication between NSX Manager and NSX Controller cluster through the User World Agent. NSX Manager sends logical interface information to the Control VM and NSX Controller cluster, and the Control VM sends routing updates to the NSX Controller cluster.



Page 27 of 130

Figure 13. NSX for vSphere Universal Distributed Logical Router

Designated Instance

The designated instance is responsible for resolving ARP on a VLAN LIF. There is one designated instance per VLAN LIF. The selection of an ESXi host as a designated instance is performed automatically by the NSX Controller cluster and that information is pushed to all other hosts. Any ARP requests sent by the distributed logical router on the same subnet are handled by the same host. In case of host failure, the controller selects a new host as the designated instance and makes that information available to other hosts.

User World Agent

User World Agent (UWA) is a TCP and SSL client that enables communication between the ESXi hosts and NSX Controller nodes, and the retrieval of information from NSX Manager through interaction with the message bus agent.

Edge Service Gateways

While the Universal Logical Router provides VM to VM or east-west routing, the NSX Edge Service Gateway provides north-south connectivity, by peering with upstream Top of Rack switches, thereby enabling tenants to access public networks.

Logical Firewall

NSX for vSphere Logical Firewall provides security mechanisms for dynamic virtual data centers.

The Distributed Firewall allows you to segment virtual data center entities like virtual machines. Segmentation can be based on VM names and attributes, user identity, vCenter objects like data centers, and hosts, or can be based on traditional networking attributes like IP addresses, port groups, and so on.

The Edge Firewall component helps you meet key perimeter security requirements, such as building DMZs based on IP/VLAN constructs, tenant-to-tenant isolation in multi-tenant virtual data centers, Network Address Translation (NAT), partner (extranet) VPNs, and user-based SSL VPNs.



Page 28 of 130

The Flow Monitoring feature displays network activity between virtual machines at the application protocol level. You can use this information to audit network traffic, define and refine firewall policies, and identify threats to your network.

Logical Virtual Private Networks (VPNs)

SSL VPN-Plus allows remote users to access private corporate applications. IPSec VPN offers site-to-site connectivity between an NSX Edge instance and remote sites. L2 VPN allows you to extend your datacenter by allowing virtual machines to retain network connectivity across geographical boundaries.

Logical Load Balancer

The NSX Edge load balancer enables network traffic to follow multiple paths to a specific destination. It distributes incoming service requests evenly among multiple servers in such a way that the load distribution is transparent to users. Load balancing thus helps in achieving optimal resource utilization, maximizing throughput, minimizing response time, and avoiding overload. NSX Edge provides load balancing up to Layer 7.

Service Composer

Service Composer helps you provision and assign network and security services to applications in a virtual infrastructure. You map these services to a security group, and the services are applied to the virtual machines in the security group.

Data Security provides visibility into sensitive data that are stored within your organization's virtualized and cloud environments. Based on the violations that are reported by the NSX for vSphere Data Security component, NSX security or enterprise administrators can ensure that sensitive data is adequately protected and assess compliance with regulations around the world.

NSX for vSphere Extensibility

VMware partners integrate their solutions with the NSX for vSphere platform to enable an integrated experience across the entire SDDC. Data center operators can provision complex, multi-tier virtual networks in seconds, independent of the underlying network topology or components.

2.2.4 Logging Architecture

vRealize Log Insight provides real-time log management and log analysis with machine learning-based intelligent grouping, high-performance searching, and troubleshooting across physical, virtual, and cloud environments.

2.2.4.1. Overview

vRealize Log Insight collects data from ESXi hosts using the syslog protocol. It connects to vCenter Server to collect events, tasks, and alarms data, and integrates with vRealize Operations Manager to send notification events and enable launch in context. It also functions as a collection and analysis point for any system capable of sending syslog data. In addition to syslog data an ingestion agent can be installed on Linux or Windows servers to collect logs. This agent approach is especially useful for custom logs and operating systems that don't natively support the syslog protocol, such as Windows.

2.2.4.2. Installation Models

You can deploy vRealize Log Insight as a virtual appliance in one of the following configurations:

Standalone node

Highly available cluster of one master and at least two worker nodes using an internal load balancer (ILB)

The compute and storage resources of the vRealize Log Insight instances for scale-up.



Page 29 of 130

2.2.4.3. Cluster Nodes

For high availability and scalability, you can deploy several vRealize Log Insight instances in a cluster where they can have either of the following roles:

Master Node. Required initial node in the cluster. The master node is responsible for queries and log ingestion. The Web user interface of the master node serves as the single pane of glass for the cluster. All queries against data are directed to the master, which in turn queries the workers as appropriate.

Worker Node. Enables scale-out in larger environments. A worker node is responsible for ingestion of logs. A worker node stores logs locally. If a worker node is down, the logs on that worker becomes unavailable. You need at least two worker nodes to form a cluster with the master node.

Integrated Load Balancer (ILB). Provides high availability (HA). The ILB runs on one of the cluster nodes. If the node that hosts the ILB Virtual IP (VIP) address stops responding, the VIP address is failed over to another node in the cluster.

2.2.4.4. Architecture of a Cluster

The architecture of vRealize Log Insight enables several channels for HA collection of log messages.

Figure 14. Cluster Architecture of vRealize Log Insight

vRealize Log Insight clients connect to ILB VIP address and use the Web user interface and ingestion (via Syslog or the Ingestion API) to send logs to vRealize Log Insight.

By default, the vRealize Log Insight Solution collects data from vCenter Server systems and ESXi hosts. For forwarding logs from NSX for vSphere, and vRealize Automation, use content packs which contain extensions or provide integration with other systems in the SDDC.



Page 30 of 130

2.2.4.5. Integration with vRealize Operations Manager

The integration with vRealize Operations Manager provides a single pane of glass for monitoring the SDDC. vRealize Log Insight sends notification events to vRealize Operations Manager. You can also launch vRealize Log Insight from the vRealize Operations Manager Web user interface.

2.2.4.6. Archiving

vRealize Log Insight supports data archiving on NFS shared storage that each vRealize Log Insight node can access.

2.2.4.7. Backup

You back up each vRealize Log Insight cluster locally by using traditional virtual machine backup solutions, such as a vSphere Storage APIs for Data Protection (VADP) compatible backup software like vSphere Data Protection.

2.2.4.8. Multi-Region vRealize Log Insight Deployment

The scope of the SDDC design covers multiple regions. Using vRealize Log Insight in a multi-region design can provide a syslog infrastructure in all regions of the SDDC. Using vRealize Log Insight across multiple regions requires deploying a cluster in each region. vRealize Log Insight supports event forwarding to other vRealize Log Insight deployments across regions in the SDDC. Implementing failover by using vSphere Replication or disaster recovery by using Site Recovery Manager is not necessary. The event forwarding feature adds tags to log message that identify the source region and event filtering prevents looping messages between the regions.

2.2.5 Operations Management Architecture

vRealize Operations Manager tracks and analyzes the operation of multiple data sources within the Software-Defined Data Center (SDDC) by using specialized analytics algorithms. These algorithms help vRealize Operations Manager to learn and predicts the behavior of every object it monitors. Users access this information by using views, reports, and dashboards.

2.2.5.1. Installation Models

vRealize Operations Manager is available in two different deployment models - a preconfigured virtual appliance, or a Windows or Linux installable package. Select the installation method according to the following considerations:

When you use the vRealize Operations Manager virtual appliance, you deploy the OVF file of the virtual appliance once for each cluster node. You access the product to set up cluster nodes according to their role, and log in to configure the installation.

Use virtual appliance deployment to easily create vRealize Operations Manager nodes with pre-defined identical size.

When you use the Windows or Linux installable package, you run the vRealize Operations Manager installation on each cluster node. You access the product to set up cluster nodes according to their role, and log in to configure the installation.

Use installable package deployment to create vRealize Operations Manager node with custom identical size.

2.2.5.2. Architecture

vRealize Operations Manager contains functional elements that collaborate for data analysis and storage, and support creating clusters of nodes with different roles.



Page 31 of 130

Figure 15. vRealize Operations Manager Architecture

2.2.5.3. Types of Nodes and Clusters

For high availability and scalability, you can deploy several vRealize Operations Manager instances in a cluster where they can have either of the following roles:

Master Node. Required initial node in the cluster. In large-scale environments the master node manages all other nodes. In small-scale environments, the master node is the single standalone vRealize Operations Manager node.

Master Replica Node. (Optional) Enables high availability of the master node.

Data Node. Enables scale-out of vRealize Operations Manager in larger environments. Data nodes have adapters installed to perform collection and analysis. Data nodes also host vRealize Operations Manager management packs.

Larger deployments usually include adapters only on data nodes, not on the master node or replica node

Remote Collector Node. In distributed deployments, enables navigation through firewalls, interfaces with a remote data source, reduces bandwidth across regions, or reduces the load on the vRealize Operations Manager analytics cluster. Remote collector nodes only gather objects for the inventory and forward collected data to the data nodes. Remote collector nodes do not store data or perform analysis. In addition, you can install them on a different operating system than the rest of the cluster nodes.

The master and master replica nodes are data nodes with extended capabilities.

vRealize Operations Manager can form two type of clusters according to the nodes that participate in a cluster:

Analytics clusters. Tracks, analyzes, and predicts the operation of monitored systems. Consists of a master node, data nodes, and optionally of a master replica node.

Remote collectors cluster. Only collects diagnostics data without storage or analysis. Consists only of remote collector nodes.



Page 32 of 130

2.2.5.4. Application Functional Components

The functional components of a vRealize Operations Manager instance interact to provide analysis of diagnostics data from the data center and visualize the result in the Web user interface.

Table 1. vRealize Operations Manager Logical Node Architecture

Architecture Component Diagram Description

Admin / Product UI server. The UI server is a Web application that serves as both user and administration interface.

REST API / Collector. The Collector collects data from all components in the data center.

Controller. The Controller handles the data flow the UI server, Collector, and the analytics engine.

Analytics. The Analytics engine creates all associations and correlations between various data sets, handles all super metric calculations, performs all capacity planning functions, and is responsible for triggering alerts.

Persistence. The persistence layer handles the read and write operations on the underlying databases across all nodes.

FSDB. The File System Database (FSDB) stores collected metrics in raw format. FSDB is available in all the nodes.

xDB (HIS). The xDB stores data from the Historical Inventory Service (HIS). This component is available only on the master and master replica nodes.

Global xDB. The Global xDB stores user preferences, alerts, and alarms, and customization that is related to the vRealize Operations Manager. This component is available only on the master and master replica nodes

2.2.5.5. Management Packs

Management packs contain extensions and third-party integration software. They add dashboards, alerts definitions, policies, reports, and other content to the inventory of vRealize Operations Manager. You can learn more details about and download management packs from the VMware Solutions Exchange.

2.2.5.6. Multi-Region vRealize Operations Manager Deployment

The scope of the SDDC design covers multiple regions. Using vRealize Operations Manager across multiple regions requires deploying an analytics cluster that is protected by Site Recovery Manager, and deploying remote collectors in each region.



Page 33 of 130

3. Detailed Design

This Software-Defined Data Center (SDDC) detailed design consists of the following main sections:

Physical Infrastructure Design

Virtual Infrastructure Design

Cloud Management Platform Design

Operations Infrastructure Design

Each section is divided into subsections that include detailed discussion, diagrams, and design decisions with justifications.

The Physical Infrastructure Design section focuses on the three main pillars of any data center, compute, storage and network. In this section you find information about availability zones and regions. The section also provides details on the rack and pod configuration, and on physical hosts and the associated storage and network configurations.

In the Virtual Infrastructure Design section, you find details on the core virtualization software configuration. This section has information on the ESXi hypervisor, vCenter Server, the virtual network design including VMware NSX, and on software-defined storage for VMware Virtual SAN. This section also includes details on business continuity (backup and restore) and on disaster recovery.

The Cloud Management Platform Design section contains information on the consumption and orchestration layer of the SDDC stack, which uses vRealize Automation and vRealize Orchestrator. IT organizations can use the fully distributed and scalable architecture to streamline their provisioning and decommissioning operations.

The Operations Infrastructure Design section explains how to architect, install, and configure vRealize Operations Manager and vRealize Log Insight. You learn how to ensure that service management within the SDDC is comprehensive. This section ties directly into the Operational Guidance section.

3.1 Physical Infrastructure Design

The physical infrastructure design includes details on decisions covering availability zones and regions and pod layout within data center racks. This section then goes on to detail decisions related to server, networking, and storage hardware.

Figure 16. Physical Infrastructure Design



Page 34 of 130

3.1.1 Physical Design Fundamentals

The SDDC physical layer design is based on a pod architecture. The physical data center elements include racks, servers, network elements, and storage arrays.

Figure 17. Physical Layer within the SDDC

3.1.1.1. Availability Zones and Regions

Availability zones and regions are used for different purposes.

Availability zones. An availability zone is the fault domain of the SDDC. Multiple availability zones can provide continuous availability of an SDDC, minimize unavailability of services and improve SLAs.

Regions. Regions provide disaster recovery across different SDDC instances. This design uses two regions. Each region is a separate SDDC instance. The regions have a similar physical layer design and virtual infrastructure design but different naming. For information on exceptions to this design, see the Business Continuity / Disaster Recovery Design chapter.

Note This design leverages a single availability zone for a one region deployment, and a single availability zone in each region in the case of a two region deployment.

The design uses the following regions. The region identifier uses United Nations Code for Trade and Transport Locations (UN/LOCODE) along with a numeric instance ID.

Table 2. Regions

Region Region Identifier

Region-specific Domain Name

Region Description

A SFO01 sfo01.rainpole.local San Francisco, CA, USA based data center

B LAX01 lax01.rainpole.local Los Angeles, CA, USA based data center



Page 35 of 130

Table 3. Availability Zones and Regions Design Decisions

Decision ID

Design Decision Design Justification Design Implication

SDDC-PHY-001

Per region, a single availability zone that can support all SDDC management components is deployed.

A single availability zone can support all SDDC management and compute components for a region. You can later add another availability zone to extend and scale the management and compute capabilities of the SDDC.

Results in limited redundancy of the overall solution. The single availability zone can become a single point of failure and prevent high-availability design solutions.

SDDC-PHY-002

Use two regions. Supports the technical requirement of multi-region failover capability as outlined in the design objectives.

Having multiple regions will require an increased solution footprint and associated costs.

3.1.1.2. Pods and Racks

The SDDC functionality is split across multiple pods. Each pod can occupy one rack or multiple racks. The total number of pods for each pod type depends on scalability needs.

Figure 18. SDDC Pod Architecture



Page 36 of 130

Table 4. Required Number of Racks

Pod (Function) Required Number of Racks

(for full scale deployment)

Minimum Number of Racks

Comment

Management pod and shared edge and compute pod

1 1 Two half-racks are sufficient for the management pod and shared edge and compute pod. As the number and resource usage of compute VMs increase adding additional hosts to the cluster will be required, as such extra space in

vmware validated design reference architecture...

Documents