Clusterware Networking When designing your cluster architecture you will need to consider the physical networking that will be required. You will need to consider the network adapters, high availability architectures, use of jumbo frames and assignment of IP addresses. Let’s look at each of these topics in more detail. 

 Network AdaptersEach server to be used in an Oracle Clusterware configuration will require a minimum of two separate network components. These two adapters are

The public network adapter  The private cluster interconnect

Public Network AdapterThe first is the public network adapter.  The public network adapter must support TCP/IP and is generally connected to the local internet via a switch. Administrators and users will connect to the database via the public network adapter. Many servers have more than one public network adapter and these are often “bonded”. Bonding provides the ability to combine each public network adapter into one logical network adapter. Bonding provides more throughput and thus better performance. Performance on the public network is quite important for the overall performance of the database. High performing networks make for faster file transfers (for example backups or export/import) and faster transfer of large amounts of application data.Private Network AdapterThe second required network adapter is required to act as the cluster interconnect. The cluster interconnect should support TCP/IP and UDP (when using UNIX or Windows platforms). The cluster interconnect is a local connection between the individual servers, and should not be part of a public network. The public interconnects are for local traffic between the clustered servers. Because the private interconnect traffic should not mix with public traffic then the private interconnect connections should have their own switches. Cross-over cables are not supported, and you must ensure that your configuration is supported by Oracle. Bonded interfaces are supported for improved throughput.Once you have configured Oracle Clusterware and installed RAC databases you may wish to confirm that the RAC databases are using the correct adapter for the Private Network Adapter. You can use the V$CLUSTER_INTERCONNECT view from any node of the RAC database. You can also use the Oracle oradebug command to determine this information. You can access oradebug through the Oracle SQL*Plus utility. You can then use the oradebug setmypid and oradebug ipc commands to determine if the correct adapter is being used for the private interconnect. Here is an example:/u01>sqlplus / as sysdbaSQL>  ORADEBUG SETMYPIDSQL> oradebug ipc

No Single Points of Failure PleaseIt is very important to avoid a single point of failure when it comes to your networking. This includes making sure you have redundant interfaces, redundant switches, redundant everything to ensure that no one piece can break.Administering RAC Network SettingsOracle Clusterware/RAC administrators will want to be able to monitor and administer network settings related to the cluster. Tools like oifcfg and svrctl are available for these tasks.Using oifcfg you can:

List the interfaces available to the cluster as seen in this example:

Oifcfg iflist –p -n Determine the public and private interfaces that have been configured for use as seen in this example:

Oifcfg getif

You can use the srvctl command to determine the VIP hostname, address, subnet mask and interface name as seen in this example:Srvctl config nodeapps -a

Changing the Private Interconnect AdapterRecall that all RAC Clusters have at least two types of network adapters. The first is the public adapter and the second is the private interconnect. You may have cause to change the specification for the interconnect adapter (such as changing the IP address). Here are the instructions to do just that:

1. Log-on to one node of the cluster. Add the new global interface specification using the oifcfg command as seen here:

Oifcfg setif –global eth2/192.0.2.0:cluster_interconnect2. Check the changes to make sure they took effect with the oifcfg command as seen here:

Oifcfg getif3. Stop the cluster using the crsctl command. Execute this on each node of the cluster:

Crsctl stop crs4. On each node, use the ifconfig command to assign the network address to the adapters:

Ifconfig eth2 192.0.2.15 netmask 255.255.255.0 broadcast 192.0.2.2555. Remove the previous adapter specification using the oifcfg command as seen in this example:

Oifcfg delif –global eth1/192.168.1.0Do not execute this step until you are sure that the replacement interface has been properly added into your cluster.6. Restart the Clusterware using the crsctl command:

Crsctl start crs 

High Availability Networking ArchitecturesYour networking configurations should be architected for high availability. In a best case situation, both the private interconnect and the public interconnect would have bonded, redundant network cards and redundant switches. As a reminder, the interface names on each node of the cluster (and slots) need to be the same. Thus if the private interconnect is on eth1 on node one, it needs to be on eth1 on the remaining nodes. Jumbo FramesEthernet packages network messages in “frames” which can be of a variable size. The frame size is called the MTU or the maximum transmission unit. When a message is larger than the MTU size (typically 1500 bytes), then it is split into multiple messages. This splitting of messages incurs additional overhead and network traffic. This can lead to RAC performance problems. In Oracle two main factors can influence the maximum size of a message:

DB_BLOCK_SIZE * DB_FILE_MULTI_BLOCK_READ_COUNT (MBRC) impacts the maximum size of a message for the global cache.  For example, a DB_BLOCK_SIZE of 8k and a MBRC of 32 will result in a maximum message size of 256k. This will result in approximately 170 separate packets. Using Jumbo frames (9k packet sizes) would result in only approximately 28 packets.  

PARELLEL_EXECUTION_MESSAGE_SIZE (defaults to 16384 bytes) determines the maximum size of a message used in parallel execution. These messages can range from 2k to over 64k.

One solution to these issues is to configure the cluster interconnect to use Jumbo Frames. When you configure Jumbo Frames the Ethernet frame size can be as large as 9k in size. Configuring for Jumbo Frames requires some careful configuration. The following steps are required to configure jumbo frames (these steps might vary based on your hardware and operating system):

Check Oracle Metalink for more information on implementing Jumbo Frames. Check for information specific to your hardware, operating system and any bugs that might exist.

Configure the host's network adapter with a persistent MTU size of 9000. On a Unix system you might use the command ifconfig -mtu 9000.

Check your vendor NIC configuration requirements. The NIC may well require some configuration.

Ensure your switches will support the larger frame size and configure the LAN switches to increase the MTU for Jumbo Frame support.

You can use traceroute for some basic configuration testing as seen in this example where we do a traceroute with a 9k packet size and a 9001 byte packet size:

traceroute –F linux01.myhost.com 9000traceroute –F linux01.myhost.com 9001Note that Jumbo Frames do not have any standard that they adhere to. As a result Jumbo Frames the interoperability between switches can be problematic and can require advanced networking skills to troubleshoot. Also keep in mind that the smallest MTU used by any device in a given network path will determine the maximum MTU for all traffic travelling along that path. As with all changes make sure that you completely test your configuration before implementing it in production. Assignment of IP AddressesBefore installing Oracle Clusterware you will want to define the public and private adapter IP addresses. There are two options you can choose with respect to IP address assignment. You can choose static IP addresses for each adapter which will require the assignment of several static IP addresses. Alternatively you can use Oracle Grid Naming Service (GNS) which will require the assignment of one static IP address. GNS will then assign dynamic IP addresses to the nodes of the cluster.When installing Oracle Clusterware you will be prompted to identify the private and public adapters.  Also you should be aware if you perform any maintenance or change the network adapters that the same adapter name (for example ETH1) should be used for the public and private adapters on each server.When assigning the IP address for the private adapter you need to make sure that you use a non-routable IP address, and make sure that all interfaces are configured for the correct speed, duplex and so on. The VIP, GNS and Single Client Access Name (SCAN)Several networking components are associated with Oracle Clusterware and Oracle RAC. These include the VIP, the GNS and the SCAN. The VIPThe purpose of the Virtual IP (VIP) is to provide for high availability. A VIP address is assigned to a given node. When that node goes down, the VIP is moved to one of the surviving nodes. This new VIP is re-arped. As a result new connections will get sent to a surviving node, and connections connected to the failed node will receive an error and attempt to re-connect (assuming the application handles the errors gracefully which can be a problem). For clients connected to the failed node, this can speed up the timeout much quicker since they don’t have to wait for a TCP/IP timeout, thus facilitating a quicker failover.When assigning an IP address to the VIP make sure that it is on the same subnet as the default gateway. Grid Naming Service (GNS)GNS is essentially a DNS for the Oracle Cluster. GNS is an optional networking component of Oracle Clustereware 11g Release 2 that provides for dynamic name resolution in the cluster. GNS removes the need to have static IP addresses assigned to the network nodes.GNS also removes the need to request VIPs if the cluster changes. Instead only one static IP is required which is the GNS virtual IP address. In conjunction with your DHCP server, GNS will

then assign dynamic IP addresses/VIPs to the cluster nodes as required. The end result is GNS makes it easier to add and remove cluster nodes. SCANOracle Clusterware 11g Release 2 introduced the Single Client Access Name (SCAN). The SCAN address provides a single address for clients to use when accessing the cluster. Before SCAN, if the cluster changed, then the client TNSNAMES.ORA files (or other tns connect strings like Easy Connect strings) would need to change. The SCAN is a fully qualified domain name that typically resolves to three separate IP addresses.  The SCAN provides a stable address that does not change regardless of the number of nodes on the cluster. The SCAN also provides for a highly available address, much like the VIPs. The SCAN simplifies network connectivity for applications using Easy Connect strings or JDBC thin client connection strings.If you are using GNS, then you have DHCP support, then the GNS will assign the addresses to the SCAN dynamically. If you are not using GNS then the SCAN will be defined in the DNS server and access requests will be resolved to one of three different IP addresses by the DNS. These addresses resolve to the SCAN listener, and the SCAN listener will resolve the connect request to one of the nodes of the cluster. The SCAN listener will load balance the connection requests to the least busy cluster as they come in.The following diagram displays the relationships between GNS, SCAN, Listeners and the Oracle Cluster and its associated databases.

 Administering SCAN ResourcesAs an Clusterware/RAC administrator you may need to administer SCAN resources on your cluster. This can include:

Adding a SCAN VIP resource.

Use the srvctl command to add a SCAN VIP resource as seen in this example where we add a new resource called rac03-scan. Note that this command will create the same number of SCAN VIP resources as there are the number of IP addresses that SCAN resolves to. If you are using DHCP or GPNP then Oracle will configure 3 SCAN VIP’s. Other options in the command allow you to define the network number, subnet and so on:

Srvctl add scan –n rac03-scan Removing a SCAN VIP resource.

Use the srvctl command to remove an existing SCAN VIP from the cluster as seen in this example:Srvctl remove scan -f

You may want to add a SCAN listener resource. To do so use the srvctl command as seen in the following example. Note in the example that we are assigning the listener to a non-default port:

Srvctl add scan_listener –p 65001 You may need to remove Clusterware resources from the SCAN listeners. Use the srvctl command to

perform this activity as seen here:

Srvctl remove scan_listener -f You may need to modify the SCAN VIP to that it matches another SCAN VIP. To do so use srvctl again as

seen here:

modify scan –n new_scan_name If you have changed the SCAN VIP configuration you will need to update the SCN listeners too. Use the

srvctl command to perform this action:

Srvctl modify scan_listener -u After making changes to the SCAN listener, you can verify those changes with the srvctl command as seen

in this example:

Srvctl config scan_listener