Exanodes 5.0.3 for VMware

User's Guide

Exanodes 5.0.3 for VMware: User's Guide

vm-5.0.3-20110531

Publication date 2011-05-27Copyright © 2011 Seanodes

Abstract

This Guide contains information on how to administer the Exanodes application. It is a step-by-step,task-oriented guide for day to day administration of the Exanodes solution.

Disclaimer and Copyright

The information contained in this publication is subject to change without notice. SEANODES® makes no warranty of any kindwith regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particularpurpose. SEANODES shall not be liable for errors contained herein or for incidental or consequential damages in connectionwith the furnishing, performance, or use of this manual.

All Rights Reserved. SEANODES®, Exanodes®, are registered trademarks of SEANODES. Other product names mentioned hereinmay be trademarks or registered trademarks of their respective companies.

Exanodes® includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (http://www.openssl.org/).

iii

Table of Contents1. Preface ............................................................................................... 1

1.1. Support ...................................................................................... 12. Introduction ......................................................................................... 2

2.1. What is shared internal storage ......................................................... 23. Fault tolerance ..................................................................................... 4

3.1. SPOF groups ................................................................................ 43.2. Disk group layouts ......................................................................... 4

4. Administration ...................................................................................... 64.1. Overview .................................................................................... 64.2. Configuration example .................................................................... 64.3. Creating a cluster ......................................................................... 64.4. Starting a cluster .......................................................................... 74.5. Cluster status ............................................................................... 74.6. Creating and starting a disk group ...................................................... 74.7. Creating and starting a volume ......................................................... 8

4.7.1. iSCSI specifics ................................................................ 94.8. Resizing a volume ......................................................................... 94.9. Stopping a volume ....................................................................... 104.10. Deleting a volume ...................................................................... 104.11. Stopping a disk group .................................................................. 104.12. Deleting a disk group .................................................................. 114.13. Stopping a cluster ...................................................................... 114.14. Deleting a cluster ....................................................................... 114.15. Recovering a disk group ............................................................... 114.16. Reintegrating a node into a cluster .................................................. 134.17. Displaying Exanodes' state ............................................................. 13

4.17.1. OK state .................................................................... 134.17.2. DEGRADED state .......................................................... 144.17.3. REBUILDING state ......................................................... 144.17.4. USING SPARE state ........................................................ 154.17.5. OFFLINE state ............................................................. 154.17.6. Special cases .............................................................. 16

4.18. Defining SPOF groups ................................................................... 174.18.1. Why define SPOF groups? ................................................ 174.18.2. How to define SPOF groups ............................................. 174.18.3. Impact of SPOF groups ................................................... 18

5. Evolution ........................................................................................... 205.1. Upgrading Exanodes ...................................................................... 20

5.1.1. Rolling upgrade ............................................................. 20

Exanodes 5.0.3 for VMware

iv

5.1.2. Full upgrade ................................................................ 205.2. Node management ....................................................................... 21

5.2.1. Adding a node .............................................................. 215.2.2. Removing a node ........................................................... 225.2.3. Node maintenance ......................................................... 23

5.3. Disk management ........................................................................ 245.3.1. Adding a disk to a cluster ................................................ 245.3.2. Adding a disk to an existing group ...................................... 245.3.3. Removing a disk ............................................................ 255.3.4. Replacing a disk ............................................................ 25

5.4. License management .................................................................... 266. Limitations ......................................................................................... 27

6.1. Cluster limitations ........................................................................ 276.2. Node limitations .......................................................................... 276.3. Disk limitations ........................................................................... 276.4. Disk group limitations ................................................................... 27

A. Regular expansions ............................................................................... 28B. iSCSI target configuration ....................................................................... 29

B.1. Changing the listen address ............................................................ 29C. Troubleshooting ................................................................................... 30Glossary ................................................................................................ 32

1

1. PrefaceThis guide provides information on Exanodes . It is intended for Operators and Systemadministrators responsible for configuration and administration of the Exanodes solution.This guide assumes you have a basic understanding of your operating system.

For information on supported hardware and software, refer to the Exanodes 5.0.3 for VMwareRelease Notes.

For information on installation, refer to the Exanodes 5.0.3 for VMware Quick Start Guide.

For command line reference, refer to the Exanodes 5.0.3 for VMware Reference Guide.

1.1 Support

For technical assistance, registered users may contact SEANODES support via Email:<support@seanodes.com> or via the Support web site: http://support.seanodes.com.From there you can open a support incident, search the knowledge base, read white papersrelated to Exanodes and access the latest product documentation and technical notes.

For additional information about SEANODES and SEANODES products, visit http://www.seanodes.com.

2

2. Introduction2.1 What is shared internal storage

Shared Internal Storage (SIS) allows users to share disks and arrays directly attached to,or embedded in, servers as if they were part of an external array. You already know andappreciate the benefits of NAS, SAN and DAS. SIS eliminates the need to buy expensiveadditional SAN or NAS hardware.

Exanodes creates disk arrays (disk groups) from disks belonging to cluster nodes. Within diskgroups, Exanodes creates volumes. As many volumes can be created, resized or removedfrom the disk groups within the limit of the aggregate disk size.

Volumes are shared between several nodes as iSCSI LUNs.

See Figure 2.1, “Exanodes integration with ESX”, for a high level view of the Exanodes VMimplementation on a particular ESX host.

Figure 2.1. Exanodes integration with ESX

Local physical disks (1) are used for two purposes: storing the local Exanodes VM on the ESX(one and only one Exanodes VM per ESX host) and as the storage pools that the ExanodesVM will use.

Each Exanodes VM must be capable of accessing other Exanodes VMs through the network towork properly and to be able to bring the level of functionalities and fault tolerance thatExanodes offers to the storage.

Introduction

3

VMFS is made of two components: (2) VMHBA devices where data is actually stored, uponwhich (3) the datastores and virtual disks are created.

Exanodes (4) aggregates virtual disks (3) (from the VMHBA physical devices (2)). It thenvirtualizes them to create volumes that are exported through an iSCSI target (5).

The storage is then imported back in ESX through the iSCSI software initiator (6), toparticipate to the VMFS storage pool as LUNs. Once formatted as a VMFS datastore, any VMvirtual disk (7) can be created on this storage: VM data or the VM itself, to be used by theapplication VMs (8).

Figure 2.2, “A typical Exanodes VM cluster” illustrates this mechanism at the cluster level.

Figure 2.2. A typical Exanodes VM cluster

Each ESX host runs one and only one Exanodes VM. Thanks to the Shared Internal Storage,each ESX has access to the same same datastores exported by the Exanodes iSCSI target. Soany VM (like VM3) can be started on any ESX host and migrated to any ESX host.

NoteIt is mandatory that the Exanodes storage used by other VMs goes through VMFS to benefitfrom the VMFS functionalities like VMotion and snapshots. It is not recommended toconfigure a VM with direct access to the Exanodes storage via an iSCSI connection.

4

3. Fault toleranceFault tolerance mechanisms in Exanodes provide tolerance to node and disk failures.

A node failure means that one node included in the cluster was affected by failures in sucha way that it does not participate anymore to the cluster activity. In a cluster of N nodes,Exanodes tolerates the failure of nodes as long as there is a majority of up and runningnodes, i.e. at least N/2+1 nodes.

A disk failure means that one disk belonging to a node of the cluster and used to build adisk group was affected by an internal failure and can't be used anymore.

NoteThese two types of failures may appear together, and may be correlated. The node failureimplies the failure of all the disks it hosts.

3.1 SPOF groups

A Single Point Of Failure (SPOF) is a part of a system which, if it fails, prevents the systemfrom working.

A SPOF group is a group of nodes that share a common SPOF (ie, that are very likely to failsimultaneously). Any number of failures in a SPOF group has the impact of a single nodefailure.

For example, nodes in the same rack and powered by the same power supply will all go downif the power supply fails. The power supply is a common SPOF for these nodes. By declaringthat the nodes in the rack (all powered by a single power supply) are all in the same SPOFgroup, you will let Exanodes support the failure of the whole rack.

By default each node in an Exanodes cluster is a SPOF group unto itself.

SPOF groups is an advanced topic and a section is dedicated to it in Section 4.18, “DefiningSPOF groups”.

3.2 Disk group layouts

A layout represents the way Exanodes will store your data on the disk group. Consider it asa RAID level. When creating a disk group, you must choose a layout type.

Exanodes supports two types of layouts. Table 3.1, “Layouts” will help you select which oneis best for you.

Fault tolerance

5

The total number of failures tolerated within a cluster or a disk group depends on thecluster configuration (symmetric or asymmetric) and whether the failure affects a nodewhich provides disks for the disk groups.

Table 3.1. Layouts

Layout Specification

sstriping The sstriping data layout is very efficient in terms of speed but has nodata redundancy. The minimum disk requirement is one, and you canuse any number of disks per node.

rainX The RainX data layout allows a disk group to tolerate the failure of onenode (or any/all the disks it hosts) while providing full access to thestorage.

With this layout, each data block of the group is mirrored, and themirror location is computed in such a way that any node failure thatcan occur will not interrupt the storage service (i.e., the data will stillbe available to the application).

You can specify a number of spares at group creation. In case of failure,Exanodes will replicate the data from the missing disk or node to thespare space in order to have again a full duplicate of the data. Whenthe replication is complete, the disk group can tolerate the failure ofanother node.

6

4. Administration4.1 Overview

This chapter will present a basic administration scenario. To learn more about a command'susage, you may run it with option --help, refer to its man page or to the Exanodes ReferenceGuide.

NoteAll administrative commands should be run as the exanodes user (this user has SSHaccess to the Exanodes VM). Once an Exanodes VM is joined to a cluster it becomes amember node of the cluster.

NoteIt is advised to use FQDNs in all commands to avoid the pitfall of misconfigured networkname resolution. (If a host isn't able to infer the FQDN of another host from a non-fullyqualified name, Exanodes won't work properly.)

4.2 Configuration example

The configuration example in this chapter is based on a cluster of three nodes (ExanodesVMs). Each Exanodes virtual machine is "powered on", on its own ESX host.

To keep things simple, it is assumed that the three nodes have the same disks and partitions.Each nodes'

NoteNew virtual disks connected to an ExanodesVM are automatically added to the cluster.

The ESX hosts are connected to a high speed interconnect.

4.3 Creating a cluster

[exanodes@node1 ~]$ exa_clcreate mycluster --disk=node/1-3/.domain.com:all --license user.license.txt --really-createCreating the cluster mycluster:Checking nodes are not already used SUCCESSSending cluster configuration SUCCESS

You can replace mycluster by the name of your choice, this is the name given to the cluster.

The user.license.txt must contain the license that Seanodes IT provided with thesoftware.

Administration

7

4.4 Starting a cluster

The command exa_clstart will start the cluster:

[exanodes@node1 ~]$ exa_clstart mycluster Initializing the cluster (Please wait): Initializing the cluster SUCCESS

The cluster must be started in order to use most of the other commands.

4.5 Cluster status

The exa_clinfo command displays the cluster status, i.e. the statuses of Exanodes'software components, disks and disk groups:

[exanodes@node1 ~]$ exa_clinfo mycluster --unassignedCLUSTER 'mycluster': 3 NODES, 0 GROUPSNODES: node1.domain.com node2.domain.com node3.domain.comSOFTWARE: OK

DISKS: UNASSIGNED node1.domain.com:/dev/sdb 280G UP node2.domain.com:/dev/sdb 280G UP node3.domain.com:/dev/sdb 280G UP TOTAL 3 disks 840GDISK GROUPS: None created.

NoteThe exa_clinfo command provides a synthetic view of the cluster so as to keep theoutput minimal and easy to read. In the example above, we added the --unassignedoption to display the disks not yet assigned to any disk groups.

4.6 Creating and starting a disk group

To create a disk group, use the command exa_dgcreate. It is possible to start the group atthe same time it is created with the option -s. With the rainX layout it is possible to assigna number of spares with the --nb_spare option.

[exanodes@node1 ~]$ exa_dgcreate mycluster:mygroup -y rainX -i node/1-3/.domain.com:/dev/sdb -sCreating disk group 'cluster:mygroup'Disk group create: SUCCESSStarting disk group 'mycluster:mygroup'Disk group start: SUCCESS

Administration

8

In this example the created disk group is named mygroup and uses the rainX layouton the disks /dev/sdb of the nodes node1.domain.com, node2.domain.com andnode3.domain.com, with no spare disk.

To check that all the above operations have been performed successfully, use the commandexa_clinfo to display the group status:

[exanodes@node1 ~]$ exa_clinfo myclusterCLUSTER 'mycluster': 3 NODES, 1 GROUPSNODES: node1.domain.com node2.domain.com node3.domain.comSOFTWARE: OK

DISKS: mygroup OK

DISK GROUP STATUS DATA.NET LAYOUT #VOLS #PHYS LOGICAL CAPACITY DEV SIZE USED AVAIL %USED mygroup OK TCP rainX 0 3 400.0G 0.0B 400.0G 0%

A group can be started after it has been created with the command exa_dgstart:

[exanodes@node1 ~]$ exa_dgstart mycluster:mygroupStarting group 'mygroup' for cluster 'mycluster'Group start: SUCCESS

NoteThe group name is preceded by the cluster name in the commands presented above. Thissyntax is used in all group-related commands.

4.7 Creating and starting a volume

A volume is a portion of a disk group's storage space. Each volume is exported as an iSCSILUN on the nodes where it is started. Each volume (LUN) can be used as a VMFS datastore.Note that the iSCSI target port is fixed to 3260.

The command exa_vlcreate is used to create a volume. Volumes can only be created ona started group. The following example creates a 100GB volume named vol1 in the groupmygroup and exported through iSCSI:

[exanodes@node1 ~]$ exa_vlcreate mycluster:mygroup:vol1 -s 100G --export-method iSCSICreating a 100.0 G volume 'mygroup:vol1' for cluster 'mycluster'Creating volume 'mygroup:vol1': SUCCESS

To verify that vol1 has been created, run exa_clinfo with the option -l to only showthe volumes status:

Administration

9

[exanodes@node1 ~]$ exa_clinfo mycluster -lNODES: node1.domain.com node2.domain.com node3.domain.comVOLUMES: mygroup:vol1 100.0G SHARED iSCSI | LUN 0

To start a volume, use the command exa_vlstart:

[exanodes@node1 ~]$ exa_vlstart mycluster:mygroup:vol1Starting volume 'vol1' in the group 'mygroup' for cluster 'mycluster'Volume start: SUCCESS

4.7.1 iSCSI specifics

If at least one volume has been exported using the iSCSI export method, exa_clinfopresents the volume as EXPORTED and shows its LUN (logical unit number). With the option--iscsi-details it also shows the IQN of the nodes exporting the volume and the IQN ofthe initiators that are connected to them.

[exanodes@node1 ~]$ exa_clinfo mycluster -l --iscsi-detailsCLUSTER 'mycluster': 3 NODES, 1 GROUPSNODES: node1.domain.com node2.domain.com node3.domain.comVOLUMES: mygroup:vol1 100.0G SHARED iSCSI | LUN 0 EXPORTED node1.domain.com node2.domain.com node3.domain.com node1.domain.com (iqn.2004-05.com.seanodes:exanodes-node1.domain.com) <- iqn.1991-05.com.microsoft:win-pgwa0df8rk1 <- iqn.1991-05.com.microsoft:win-rtla9u6gt1i <- iqn.1994-05.com.redhat:b178688aed99 node2.domain.com (iqn.2004-05.com.seanodes:exanodes-node2.domain.com) node3.domain.com (iqn.2004-05.com.seanodes:exanodes-node3.domain.com) <- iqn.1991-05.com.microsoft:win-3jufm9t1i5a

Exanodes provides an IQN filtering feature allowing the administrator to choose whichinitiator can access the different volumes. The IQN filtering is activated and configuredusing exa_vltune on the parameters iqn_auth_accept, iqn_auth_reject andiqn_auth_mode.

See Appendix B, iSCSI target configuration on how to configure the clusterized iSCSI target.

4.8 Resizing a volume

A volume can be resized with exa_vlresize but this can be dangerous for the data storedon the volume. Please read carefully the man page of exa_vlresize before using it.

Administration

10

WarningExanodes Virtual Machine Edition 5.0.3 does NOT support volume shrinking.

4.9 Stopping a volume

To stop a volume, use the command exa_vlstop:

[exanodes@node1 ~]$ exa_vlstop mycluster:mygroup:vol1Stopping volume 'vol1' in the group 'mygroup' for cluster 'mycluster'Volume stop: SUCCESS

NoteMake sure that all Guest virtual machines using the volume have been migrated orpowered off prior to stopping the volume.

4.10 Deleting a volume

To delete a volume, use the command exa_vldelete:

[exanodes@node1 ~]$ exa_vldelete mycluster:mygroup:vol1 --really-deleteDeleting logical volume 'vol1' in the group 'mygroup' for cluster 'mycluster'Volume delete: SUCCESS

NoteA volume can be deleted only if it has been previously stopped. All the data stored onthis volume will be lost. It is advised to remove the VMFS datastore using the volumeprior to volume deletion.

4.11 Stopping a disk group

The command exa_dgstop will stop the group:

[exanodes@node1 ~]$ exa_dgstop mycluster:mygroupStopping disk group 'mygroup' for cluster 'mycluster'Group stop: SUCCESS

NoteWhen a disk group is stopped, Exanodes recursively stops the volumes it contains andremembers if and on which nodes they were started . The next time the group is started,the volumes will be started at the same places.

If you do not want Exanodes to remember these states use the --recursive option, whichwill stop the volumes on the group before stopping the group itself. At the next group start,the volumes will not go back to their previous state. The user can then start them manually.

Administration

11

4.12 Deleting a disk group

To delete a disk group, use the command exa_dgdelete:

[exanodes@node1 ~]$ exa_dgdelete mycluster:mygroup --really-deleteGroup delete: SUCCESS

NoteA disk group can only be deleted if it has been previously stopped. Even if the groupis stopped, it might still contain volumes; if this is the case, using the --recursiveoption will delete all of them. All the data stored on the disk group will be lost.

4.13 Stopping a cluster

To stop an Exanodes cluster, use the command exa_clstop:

[exanodes@node1 ~]$ exa_clstop myclusterStopping all nodes: node/1-3/.domain.comStopping requested nodes SUCCESS

NoteThe command exa_clinfo cannot return the cluster status if it has been stopped.

4.14 Deleting a cluster

To delete the cluster, use the command exa_cldelete:

[exanodes@node1 ~]$ exa_cldelete mycluster --really-deleteDeleting Exanodes cluster initialization for cluster 'mycluster'Checking nodes are ready SUCCESSDeleting the initialization file SUCCESS

NoteA cluster can be deleted only if it has been previously stopped. Even if the cluster isstopped, it might still contain groups and volumes; if this is the case, using the --recursive option will delete all of them. All data stored on the cluster will be lost.

4.15 Recovering a disk group

It may happen that one of the disks used to create a disk group "burns out", and this diskmust be replaced by a new one. The command exa_clinfo will show the disk status asBROKEN or MISSING.

At this point the disk group may consequently be in any of the following states:

• REBUILDING if the group layout is rainX and the disk just died.

Administration

12

• USING SPARE if two copies of data are accessible.

• DEGRADED if only one copy of data is accessible.

• OFFLINE if part of the data is no longer accessible.

[exanodes@node1 ~]$ exa_clinfo myclusterCLUSTER 'mycluster': 3 NODES, 1 GROUPSNODES: node1.domain.com node2.domain.com node3.domain.comSOFTWARE: OK

DISKS: mygroup node2.domain.com:3A3D2DD0:62D019EB:92ABFD36:9DFBC567 100.0G (0% Used) BROKEN

DISK GROUP STATUS DATA.NET LAYOUT #VOLS #PHYS LOGICAL CAPACITY DEV SIZE USED AVAIL %USED mygroup DEGRADED TCP rainX 0 3 400G 0.00B 400G 0%

NoteIn the above output we can see the unique identifier, i.e. uuid, of the BROKEN disk(3A3D2DD0:62D019EB:92ABFD36:9DFBC567).

To recover: stop the node, replace the disk, restart the node and use the commandexa_dgdiskrecover to make Exanodes rebuild data on the new disk.

[exanodes@node1 ~]$ exa_dgdiskrecover --disk node2.domain.com:/dev/sdb mycluster:mygroup --uuid 3A3D2DD0:62D019EB:92ABFD36:9DFBC567 --really-recoverRecovering group 'mycluster:mygroup', using disk 'node2.domain.com:/dev/sdb' for uuid ''Hit Ctrl-C if unsure. Remaining seconds : 0Disk 'node2.domain.com:/dev/sdb': SUCCESS

NoteIn order to recover back to a group that can handle a new failure, the rainX disk grouplayout policy is required.

[exanodes@node1 ~]$ exa_clinfo myclusterCLUSTER 'mycluster': 3 NODES, 1 GROUPSNODES: node1.domain.com node2.domain.com node3.domain.comSOFTWARE: OK

DISKS: mygroup node2.domain.com:3A3D2DD0:62D019EB:92ABFD36:9DFBC567 (/dev/sdb) 100G (1% Used) UPDATING 1%

DISK GROUP STATUS DATA.NET LAYOUT #VOLS #PHYS LOGICAL CAPACITY DEV SIZE USED AVAIL %USED mygroup REBUILDING TCP rainX 1 3 400G 0.00B 400G 0%

Administration

13

If the layout of the group is sstriping, a rebuild is NOT possible.

NoteIf there are several BROKEN disks in the group, it is possible to specify the one thatshould be rebuilt by indicating its UUID.

4.16 Reintegrating a node into a cluster

If a node's operating system is reinstalled or if a node is replaced, then it no longer belongsto the cluster. Once Exanodes is installed, you can reintegrate this node to the cluster usingthe command exa_clnoderecover:

[exanodes@node1 ~]$ exa_clnoderecover -n node2.domain.com mycluster --joinRecover nodes:Initializing the nodes:Node initialization: SUCCESS

NoteThis command only allows to replace one node; it cannot add a new node to the cluster.To add a new node, use exa_clnodeadd.

4.17 Displaying Exanodes' state

To get the status of Exanodes software and hardware components from all cluster nodes,use the command exa_clinfo.

4.17.1 OK state

When the cluster is up and running with no failure, all Exanodes components are fullyoperational and their status is OK:

[exanodes@node1 ~]$ exa_clinfo myclusterCLUSTER 'mycluster': 3 NODES, 1 GROUPSNODES: node1.domain.com node2.domain.com node3.domain.comSOFTWARE: OK

DISKS: mygroup OK

DISK GROUP STATUS DATA.NET LAYOUT #VOLS #PHYS LOGICAL CAPACITY DEV SIZE USED AVAIL %USED mygroup OK TCP rainX 1 3 400G 100G 300G 25%

VOLUMES: mygroup:vol1 100.0G SHARED iSCSI | LUN 0 EXPORTED node1.domain.com node2.domain.com node3.domain.com

Administration

14

4.17.2 DEGRADED state

If a failure affects one of the cluster nodes (for example node2.domain.com) in such away that this node and the disks exported for the group cannot be reached anymore, thenthe group status will be DEGRADED. This means that the Exanodes software and potentialapplications using the disk group are working correctly but another disk failure, in thisgroup, will not be tolerated by the system. The command exa_clinfo returns the followingoutput:

[exanodes@node1 ~]$ exa_clinfo myclusterCLUSTER 'mycluster': 3 NODES, 1 GROUPSNODES: node1.domain.com node2.domain.com node3.domain.comSOFTWARE: WARNING 1 node down node2.domain.com

DISKS: mygroup node2.domain.com:4AE7C3BE:F454DB0B:26D4A361:22E909B5 280.0G (0% Used) DOWN

DISK GROUP STATUS DATA.NET LAYOUT #VOLS #PHYS LOGICAL CAPACITY DEV SIZE USED AVAIL %USED mygroup DEGRADED TCP rainX 1 3 400G 100G 300G 25%

VOLUMES: mygroup:vol1 100.0G SHARED iSCSI | LUN 0 EXPORTED node1.domain.com node3.domain.com WILL EXPORT node2.domain.com

4.17.3 REBUILDING state

Once the failure is fixed (or if the layout used is rainX and there is spare space available), theaffected component (node or disk) will be reintegrated (or duplicated) within the cluster.This group state is called REBUILDING.

If the failure concerned a disk included in the disk group, when this disk comes back its datawill be synchronized (if necessary) with the other disks. Running the command exa_clinfoduring the synchronization process returns the following output:

Administration

15

[exanodes@node1 ~]$ exa_clinfo myclusterCLUSTER 'mycluster': 3 NODES, 1 GROUPSNODES: node1.domain.com node2.domain.com node3.domain.comSOFTWARE: OK

DISKS: mygroup node2.domain.com:4AE7C3BE:F454DB0B:26D4A361:22E909B5 (/dev/sdb) 280.0G (91% Used) UPDATING 0%

DISK GROUP STATUS DATA.NET LAYOUT #VOLS #PHYS LOGICAL CAPACITY DEV SIZE USED AVAIL %USED mygroup OK TCP rainX 1 3 400G 100G 300G 25%

VOLUMES: mygroup:vol1 100.0G SHARED iSCSI | LUN 0 EXPORTED node1.domain.com node2.domain.com node3.domain.com

4.17.4 USING SPARE state

With a 5 nodes cluster using the rainX layout with one free spare, if a disk or a node from thegroup goes down, the cluster will automatically rebuild the group (by replicating the data).Once data replication is finished, the group is back in a state which can again handle a newfailure without returning any IO error. This state is USING SPARE.

4.17.5 OFFLINE state

If a failure affects a disk group which does not tolerate it, the group will be OFFLINE andthe volumes attached to this group might return I/O errors.

For example, the sstriping data layout does not support the loss of a disk. Refer toSection 3.2, “Disk group layouts” for the definition of each layout. In this case the commandexa_clinfo shows the group status as OFFLINE:

Administration

16

[exanodes@node1 ~]$ exa_clinfo myclusterCLUSTER 'mycluster': 3 NODES, 1 GROUPSNODES: node1.domain.com node2.domain.com node3.domain.comSOFTWARE: WARNING 1 node down node2.domain.com

DISKS: mygroup node2.domain.com:/dev/sdb DOWN

DISK GROUP STATUS DATA.NET LAYOUT #VOLS #PHYS LOGICAL CAPACITY DEV SIZE USED AVAIL %USED mygroup OFFLINE TCP sstriping 1 3 400G 100G 300G 25%

VOLUMES: mygroup:vol1 100.0G SHARED iSCSI | LUN 0 WILL EXPORT node1.domain.com node2.domain.com node3.domain.com

In this example, the disk node2.domain.com:/dev/sdb is down, the group is offlinebut still administrable. This means that you can run exa_dg* commands on this group buttrying to access its volumes might lead to IO errors.

4.17.6 Special cases

If the command exa_clinfo is used while Exanodes has just detected or fixed a failure,Exanodes will take into account this event and react accordingly but the commandexa_clinfo might return the following error:

[exanodes@node1 ~]$ exa_clinfo myclusterERROR: Admind is already performing a command.

This behavior is normal and it means that the needed information cannot be provided yet;you might retry a few seconds later. It may also happen to get the 'busy' error while runninga command other than exa_clinfo.

The administrative part of Exanodes only accepts a single command at a time. This meansthat if one runs an exa_vlcreate, one has to wait for its completion before running anothercommand (except exa_clinfo and exa_clstats).

In some cases, at the top and bottom of an exa_clinfo, the following message can be seen:

WARNING: A recovery is in progress on your cluster.You should run again your command to get an updated view.

This is also a normal behavior which generally occurs when one node is being reintegratedor excluded from the cluster.

Administration

17

4.18 Defining SPOF groups

NoteThis section is merely an introduction to SPOF groups. Please contact your reseller forfurther information and guidance.

4.18.1 Why define SPOF groups?

What if some nodes in a cluster have a common underlying SPOF (such as a power supply ora network switch)? If the SPOF fails, all these nodes go down. If Exanodes has no knowledgeof this fact, it may decide to store all replicas of some data on these nodes and, would theSPOF fail, the data would be unavailable.

This is why Exanodes provides the ability to define SPOF groups: it makes Exanodes awarethat nodes in a SPOF group are very likely to fail at the same time and thus that the replicasof a piece of information must not all be stored within the nodes of a single SPOF group.Even if the SPOF fails and all the nodes in the SPOF group go down, some replicas are stillavailable.

SPOF groups are of interest only if data is replicated. Which means that it is worth definingSPOF groups when using the RainX layout, but there is no point in doing so when using theSimple Striping layout. (It doesn't hurt, however.)

4.18.2 How to define SPOF groups

SPOF groups are handled as follows:

• SPOF groups are defined at exa_clcreate with option --spof-group.

• When a node is added with exa_clnodeadd, that node becomes a SPOF unto itself. Atthe moment, it is not possible to add a node to an existing SPOF group.

• When a node is deleted with exa_clnodedel, that node is removed from its SPOF.

To create a 7-node cluster with nodes node1..node7, one could just write:

exa_clcreate mycluster --disk 'node/1-7/:/dev/sdb' --license user.license.txt

However, if one knows that node2, node3 and node4 are plugged into the same powersupply (SPOF), while node5 and node6 are plugged into another power supply (anotherSPOF), one could improve fault tolerance in the cluster by specifying SPOF groups as follows:

exa_clcreate mycluster --disk 'node/1-7/:/dev/sdb' --spof-group '[node1][node/2-4/][node/5-6/][node7]' --license user.license.txt

Administration

18

This would make Exanodes arrange the replicas of the user's data in such a way that, even ifa whole SPOF group fails, the data is still available to the user. For example, if both node5and node6 fail, it has no impact on data availability.

NoteWhen defining SPOF groups, not all nodes need appear in the SPOF groups. When absent,a node implicitely constitutes a separate SPOF group by itself.

For example, the previous command could be written:

exa_clcreate mycluster --disk 'node/1-7/:/dev/sdb' --spof-group '[node/2-4/][node/5-6/]' --license user.license.txt

Nodes node1 and node7 don't appear in the SPOF groups, therefore each of them constitutesa separate SPOF group.

4.18.3 Impact of SPOF groups

SPOF groups have an impact on the RainX disk groups that can be created: RainX has rulesregarding SPOFs, spares and quorum that must be satisfied.

Before we get to the rules proper, here are a couple of facts:

• The maximum number of cumulated SPOF failures a group supports is its number of sparesplus one. (At that point, the group is of course DEGRADED.)

• The "worst DEGRADED case" for a group is when its number of failed SPOFs is the maximumsupported and the number of nodes down, as a consequence, is the highest.

The two rules that must be satisfied when creating a RainX group are the following:

• Administrability rule: A group must not lose more than half of its nodes when in its worstDEGRADED case.

• Quorum rule: The cluster must not lose its quorum when the group is in its worstDEGRADED case.

Let's try to create a RainX disk group containing all 7 nodes with 1 spare:

$ exa_dgcreate -i node/1-7/:/dev/sdb -y rainX --nb_spare 1 mycluster:mygroupCreating disk group 'mycluster:mygroup':ERROR: Group would be non-administrable in worst degraded case: losing 2 SPOF groups may leave less than 4 nodes in the disk group.Disk group create: ERRORERROR: Layout constraints infringed.

As the error says, the group couldn't be created because in the worst case scenario, whichis when 2 SPOFS (number of spares + 1) fail and these SPOFs are [node/2-4/] and

Administration

19

[node/5-6/], the disk group would lose 5 nodes out of 7, which would leave less than themajority in the group (4 nodes).

And now with no spare:

$ exa_dgcreate -i node/1-7/:/dev/sdb -y rainX mycluster:mygroupCreating disk group 'toto:group':Disk group create: SUCCESS

The group could be created because in this case each node constitutes a separate SPOF, andlosing 1 SPOF (number of spares + 1) means losing 1 node, leaving 6 nodes which is morethan the majority in the group (4 nodes).

20

5. Evolution5.1 Upgrading Exanodes

There are two kinds of Exanodes upgrades:

• Rolling upgrade: The Exanodes cluster is not stopped, a rolling upgrade is performed.Please check version compatibility prior to performing as a full upgrade may be required.

• Full upgrade: The Exanodes cluster needs to be stopped to perfom the upgrade.

Exanodes upgrade system is based on the RPM Package Manager and SSH network protocol.

5.1.1 Rolling upgrade

To perform a rolling upgrade follow these steps on each node of the cluster:

1. Migrate all Guest VMs that are stored on the Exanodes datastore to another VMwareESX host.

2. Perform an exa_clnodestop for the node to upgrade.

3. Stop the Exanodes services on the VM.

[exanodes@node1 ~]$ service exanodes stop

4. Upload the new Exanodes RPM on the virtual machine (there could be several RPM filesto upload).

[exanodes@node1 ~]$ scp root@node1:~ exanodes-vm-<version>.rpm ExanodesVM_<version>.rpm 100% 6439 5.6KB/s 00:00

5. Install the new Exanodes RPM on the virtual machine (there could be several RPM filesto install).

[exanodes@node1 ~]$ rpm -vUh ~/exanodes-vm-<version>.rpm

6. Restart the Exanodes services on the VM.

[exanodes@node1 ~]$ service exanodes start

7. Perform a exa_clnodestart for the upgrade node.

8. Verify that the ESX iSCSI initiator has reconnected to the target. It may be necessary toperform a 'Rescan' on the Software iSCSI adapter.

9. Migrate the Guest VMs back to this VMware ESX server (if necessary).

5.1.2 Full upgrade

To perform a full upgrade follow these steps:

Evolution

21

1. Power Off all guest VMs stored on the Exanodes datastore.

2. Stop the Exanodes cluster using the exa_clstop command.

3. Stop the Exanodes service on all the the Exanodes VMs with service exanodes stop

[exanodes@node1 ~]$ service exanodes stop

4. Upload the new RPM on all the Exanodes VMs (there could be several RPM files to upload).

[exanodes@node1 ~]$ scp root@node1:~ exanodes-vm-<version>.rpm

5. Install the new RPM on all the Exanodes VMs (there could be several RPM files to install).

[exanodes@node1 ~]$ rpm -Uvh ~/exanodes-vm-<version>.rpm

6. Restart the Exanodes service on all the Exanodes VMs.

[exanodes@node1 ~]$ service exanodes start

7. Start the Exanodes cluster using the exa_clstart command.

8. Wait for the recovery to complete.

9. Restart the guest VMs that were stored on the Exanodes datastore.

Please refer to the above note regarding ExanodesVM static IP assignment if required.

5.2 Node management

You can add a node by either creating a new cluster (all data will be lost), or by dynamicallyadding a node to an existing cluster. In order to add nodes dynamically to a cluster, Exanodesprovides a set of commands: exa_clnode* (exa_clnodeadd, for instance).

5.2.1 Adding a node

Use the command exa_clnodeadd to add nodes to a cluster already started. The nodesadded to the cluster can export drives to Exanodes. Nodes and disks to use are defined using-n and -i options.

[exanodes@node1 ~]$ exa_clnodeadd mycluster -n node4.domain.com -i /dev/sdbAdding a node to cluster 'mycluster' SUCCESS

The node is now part of the cluster but it is not yet started. It is reported as DOWN byexa_clinfo.

Use the command exa_clnodestart to start the node, it will no longer be detected asDOWN:

[exanodes@node1 ~]$ exa_clnodestart mycluster -n node4.domain.comStarting one or several nodesStart one or several nodes SUCCESS

Evolution

22

[exanodes@node1 ~]$ exa_clinfo mycluster -sCLUSTER 'mycluster': 4 NODES, 1 GROUPSNODES: node1.domain.com node2.domain.com node3.domain.com node4.domain.comSOFTWARE: OK

5.2.2 Removing a node

To remove a node from a cluster use the commands exa_clnodestop andexa_clnodedel.

Since Exanodes is fault tolerant, you can still access your storage and administer your clusterwith one or more nodes down, as long as there is a quorum of nodes running. With Exanodes,the quorum is the majority, i.e. N/2 + 1, where N equals the number of nodes.

The consequence of the loss of a node depends on the disks it contains, the disk group layoutused and especially its status.

The following table indicates the consequences of a disk failure dependant on the disk grouplayout (assuming the initial status of the group is OK or USING SPARE).

Table 5.1. Impact of disk loss depending on layout

Layout Disk loss

sstriping The layout sstriping does not accept any disk failure.

If you remove a node that contains a disk, its disk group will becomeOFFLINE and can only be read (write access is no longer possible).

IO errors may be returned when trying to access data from the missingdisk.

rainX With the rainX layout, Exanodes tolerates at least a single disk failure(more if there is spare space available in the disk group).

If several disks are removed after all the spares are used, the diskgroup will become OFFLINE and can only be read (write access is nolonger possible).

IO errors will be returned when trying to access data whose disk andmirror are down.

NoteA node can only be stopped if it does not place a disk group in the state OFFLINE. Anode can only be deleted if it is in the DOWN state and does not provide storage to adisk group.

Evolution

23

[exanodes@node1 ~]$ exa_clnodestop mycluster -n node4.domain.comStopping one or several nodesStopping requested nodes SUCCESS

[exanodes@node1 ~]$ exa_clnodedel mycluster -n node4.domain.comRemoving node 'node3.domain.com' from cluster 'mycluster' SUCCESS

[exanodes@node1 ~]$ exa_clinfo mycluster -sCLUSTER 'mycluster': 3 NODES, 1 GROUPSNODES: node1.domain.com node2.domain.com node3.domain.comSOFTWARE: OK

5.2.3 Node maintenance

If offline maintenance is needed on one of the nodes while keeping the cluster running, thecommand exa_clnoderecover will allow to temporarily extract a node from the cluster.

In the following example, we exclude node2 from our cluster:

[exanodes@node1 ~]$ exa_clnodestop mycluster -n node2.domain.comStopping one or several nodesStopping requested nodes SUCCESS

[exanodes@node1 ~]$ exa_clinfo mycluster -sCLUSTER 'mycluster': 3 NODES, 0 GROUPSNODES: node1.domain.com node2.domain.com node3.domain.comSOFTWARE: WARNING1 node down node2.domain.com

[exanodes@node1 ~]$ exa_clnoderecover mycluster --leave --node node2.domain.comAsk the nodes to leave the cluster: SUCCESS

Perform the maintenance on node2 and when done, make it re-join the cluster and restart it:

[exanodes@node1 ~]$ exa_clnoderecover mycluster --join --node node2.domain.comRecover nodes: SUCCESSInitializing the nodes (Please wait):Node initialization: SUCCESS

[exanodes@node1 ~]$ exa_clnodestart mycluster -n node2.domain.comStarting one or several nodesStart one or several nodes SUCCESS

exa_clinfo will no longer display the node as down. As explained before, the commandexa_clnodestop will not work in all cases. For example, if the node that needsmaintenance is providing a disk to a simple striping disk group, the only way to stop thenode properly is to stop the group first.

Caution: in the case of a node replacement, the new node must:

• Run the same Exanodes release.

Evolution

24

• Have the same host name and network configuration.

• Have the same disks () than the previous node with at least the same storage capacity(more is acceptable but the additional space will not be used).

5.3 Disk management

5.3.1 Adding a disk to a cluster

New disks may be added to a working cluster with the command exa_cldiskadd:

[exanodes@node1 ~]$ exa_cldiskadd mycluster -i node3.domain.com:/dev/sddAdding disk 'node3.domain.com:/dev/sdd' to cluster 'mycluster' SUCCESS

Then use the command exa_clinfo with the --unassigned parameter to show the new(and thus unassigned) disks:

[exanodes@node1 ~]$ exa_clinfo mycluster --unassignedCLUSTER 'mycluster': 3 NODES, 1 GROUPSNODES: node1.domain.com node2.domain.com node3.domain.comSOFTWARE: OK

DISKS: mygroup OK UNASSIGNED node3.domain.com:/dev/sdd 280G UP TOTAL 1 disks 280GDISK GROUP STATUS DATA.NET LAYOUT #VOLS #PHYS LOGICAL CAPACITY DEV SIZE USED AVAIL %USED mygroup OK TCP rainX 0 3 400G 0.0B 400G 0%

NoteThe new disks are added to the cluster, not to any group. You can then either use theadded disks in a new group, or add them to an existing group using the exa_dgdiskaddcommand, to increase the group capacity.

5.3.2 Adding a disk to an existing group

Unassigned disks can be added to existing group when the group is stopped.

[exanodes@node1 ~]$ exa_dgdiskadd mycluster:mygroup -i node3.domain.com:/dev/sddAdding disk node3.domain.com:/dev/sdd in group 'mycluster:mygroup' SUCCESS

You can then restart the group. Depending on the group's previously used capacity, the newlyadded disk's capacity may not be immediately available.

Evolution

25

• If the group was completely used by volumes, the new disk will not be useable by itself,because Exanodes can't replicate its data across others disks on different SPOF groups. Inthis case, you have to add more disks to the group, until there are enough disks to createslots matching the group's slot width.

• If the group had free capacity, part or all of the new disk's space is immediately usable,depending on the replication rule.

5.3.3 Removing a disk

Only unassigned disks, which are not part of any group can be removed.

[exanodes@node1 ~]$ exa_cldiskdel mycluster -i node3.domain.com:/dev/sddRemoving disk 'node3.domain.com:/dev/sdd' from cluster 'mycluster' SUCCESS

5.3.4 Replacing a disk

The following table indicates how to replace a disk, depending on the disk group layout.

Table 5.2. Disk replacement depending on layout

Layout Replacement process

sstriping The sstriping layout does not accept any disk failure.

If a disk has failed, the disk group has been set to readonly. You may tryto manually copy your remaining data out of the disk group.

Then replace the disk with a new one, and recreate the disk group withexa_dgcreate.

rainX A single failure at a time is handled by this layout.

• A failure can be a disk failure, a node failure, or multiple nodefailures inside a single SPOF group.

• If the disk group became OFFLINE because too many disks wentdown, you may try to manually copy your remaining data off the diskgroup and then recreate the group with new drives.

• If the disk group is still operational, then the disk can be replaced bya new disk, with at least the same capacity.

Last, recover the disk and rebuild its data with the commandexa_dgdiskrecover

Evolution

26

5.4 License management

You can change the license used on your cluster's nodes using the exa_cllicensecommand. This allows upgrading from an Evaluation license to a Full license, or upgradingthe number of nodes allowed, for example. This command must be issued on a stoppedcluster, in which all nodes are up and reachable:

[exanodes@node1 ~]$ exa_cllicense mycluster --set-license user.license.txtChecking nodes are ready SUCCESSPushing new license on the nodes SUCCESS

27

6. LimitationsThe following provides information on Exanodes software limitations.

6.1 Cluster limitations

An Exanodes cluster can have from 1-32 member nodes, and from 0-512 disks.

An Exanodes cluster can provide at maximum 256 LUNs.

6.2 Node limitations

A node can provide to the Exanodes cluster 0-43 disks.

6.3 Disk limitations

Each disk given to Exanodes can be at most 2TB. This is a hard ESX limit.

6.4 Disk group limitations

Simple striping disk groups can contain 1-512 disks and 0-255 volumes. A Simple striping diskgroup's maximum capacity is 1024TB.

RainX disk groups can contain 3-512 disks and 0-255 volumes. A RainX disk group requires atminimum 3 nodes. A RainX disk group's maximum capacity is 512TB.

RainX disk groups must follow the following rules:

• Replication rule: The slot width must be superior or equal to the number of spares + 2:slot_width >= 2 + nb_of_spares

• Spare rule: No more than 16 spares.

28

Appendix A. Regular expansionsIn order to manage large clusters, it is often useful to be able to address a set of nodes witha single command. Exanodes recognizes so-called "regular expansions" to address a set ofnodes using range definitions.

Regular expansions are usually used for hostnames but not only. When a command allows theuse of a regular expansion it is noted in its manual page.

A regular expansion is detected by Exanodes if it is enclosed between / (slash) and containsonly numbers, the characters - (dash) and/or : (colon).

• The - character specifies a range. For example, node/1-10/ means node1 to node10.

• The : character specifies a set. For example node/1:10/ means node1 and node10.

• Both characters can be combined in a regular expansion. For example, node/1-10:15/means node1 to node10 and node15.

• Several regular expansions can be given at the same time. For examplenode/1-2/-/3-4/ means node1-3, node1-4, node2-3 and node2-4.

• A suffix can be provided: node/1-2/-lab1 defines node1-lab1 and node2-lab1.

• If leading zeros are defined in a number, they will be respected in the result. For example,node/01-03/ means node01, node02 and node03.

Other examples:

• node/1-4/ = node1 node2 node3 node4

• node-/3-5:7:10-12/ = node-3 node-4 node-5 node7 node-10 node-11 node-12

Exanodes' regular expansions can be used in scripts thanks to the commands exa_expandand exa_unexpand:

• exa_expand takes a list of regular expansions and returns the expanded list. For exampleexa_expand node/1-2/ returns node1 node2.

• exa_unexpand does the opposite, it takes a list and tries to compress it using regularexpansion rules. For example, exa_unexpand node1 node2 returns node/1-2/.

29

Appendix B. iSCSI target configurationB.1 Changing the listen address

By default, each instance of Exanodes' clusterized target listens on all addresses. However,this may be changed on a per target instance (per node) basis.

To change the listen address of Exanodes' target instance on a given node, edit the targetconfiguration file /etc/exanodes/target.conf (create it if it doesn't exist) and add aline as follows:

listen_address=IP

where IP is the IP address on which to listen. This IP address may be different on each node.

NoteIf parameter listen_address is not present in the target configuration file, or if theIP specified in that file is 0.0.0.0, or if the file is not present, the target listens on alladdresses1.

1A maximum of 10 IP addresses per node is reported.

30

Appendix C. TroubleshootingC.1. Where are Exanodes logs and traces ?

Exanodes logs its messages to /var/log/exanodes/exanodes.log. This file willcontain informational, warning and error messages. In addition to that, Exanodes logsfatal errors to the host's syslog, which is visible in /var/log/syslog on RedHat-based distributions. These messages could help you determine a problem, and will helpSeanodes' support fix problems.

C.2. "Rescan all disks" leads to an Exanodes crash

When it performs a full disk rescan, the Exanodes VM is frozen for a while. Thiscauses the other nodes to detect it as DOWN. To alleviate this scenario you can tryincreasing the heartbeat values of Exanodes. Please refer to the Exanodes' referenceguide exa_cltune section for further information.

C.3. Suspending the Exanodes VM leads to a crash

To support failure in a clustered manner, Exanodes sends heartbeats over the networkand checks heartbeats from other Exanodes VMs. This is why the Exanodes VM doesnot support to be suspended: other Exanodes instances declare it as DOWN. You shouldnever suspend any Exanodes VM when the cluster is running.

C.4. Snapshotting the Exanodes VM leads to issues

Snapshotting the Exanodes VM leads to having the cluster in a strange state afterthe snapshot is reverted. Exanodes needs certain configuration files up to date whichcannot be obtained with a snapshot. This VMware feature is not supported with theExanodes VM. However, there is absolutely no problem using snapshots with VMs on thestorage created with Exanodes.

C.5. System logs report 'NETDEV WATCHDOG: vswif0: transmit timed out'

The appliances using Exanodes storage receive IO failure. You MUST configure adedicated Virtual Network for Exanodes VM on a dedicated NIC. Then reboot theExanodes VM.

C.6. LUNs larger than 2TB do not appear

This is a known limitation of the ESX software initiator. Create LUNs smaller than 2TB.

C.7. Using templates to deploy Exanodes VM

Create the ExanodesVM templates before configuring Exanodes. Deploying from anExanodesVM that has been configured as part of an Exanodes cluster will lead to a crash.

C.8. Storage has bad performances with heterogeneous configurations

You should avoid building groups of disks with heterogeneous sizes or very differentperformance or with an unbalanced distribution on the nodes. A balanced andhomogeneous configuration avoids contention on any element of your cluster.

C.9. Shrinking a volume

Troubleshooting

31

Shrinking a VMFS formatted volume is not supported in the current version of Exanodes5.0.3.

C.10.New LUN size does not appear when a volume is grown

You should perform a full rescan of all targets on all ESX hosts to see the new size.Then go to the appropriate VMware documentation for information on how to use thisnew storage space.

C.11.Error "the hard disk is not blank" during 'Add Storage' wizard

When creating a VMFS datastore on a volume that previously held a datastore that wasnot deleted, the above message is returned during the 'Add Storage' wizard. This is dueto the fact that the volume contains the old VMFS information. As per the indicationsin Section 4.10, “Deleting a volume” it is required to delete the datastore prior todeleting the Exanodes volume.

C.12.VMware HA no longer works since configuring the Exanodes networks

This problem can be due to the IP assigned to the "iSCSI Service Console" port. To resolveyou need to assign a unique IP address to each "iSCSI service console" port accross yourESX hosts.

C.13.Exanodes VM doesn't get an IP at boot

This may come from the VM network you setup for the 2 virtual NICs. Be sure that thevNIC that uses the "flexible" driver is on the iSCSI network, and that the vNIC that usesthe "VMXNET" driver is connected to a Virtual Network that has access to a DHCP server.

32

GlossaryCluster A group of interconnected servers (nodes).

Node A system belonging to a cluster.

Disk A disk is a storage system able to read or write data toa storage device: hard drive, partition, volume, ram disk,network drive... It can be accessed as a block device from theoperating system.

Disk group A disk group is a storage subsystem that creates a storage array.A disk group agglomerates the storage space from all of itsdisks.

Volume A volume is a virtual storage area. The storage space of adisk group is used to create volumes up to the total physicalcapacity of the group.

Layout The layout defines the way Exanodes will arrange (locate) thedata on a disk group.

RainX RainX is a disk group data layout where redundant informationis split between the disks, allowing for the loss of one diskwhile providing full access to data. Moreover, it has optionalsupport for spare disks. Each spare disk allows the RainX layoutto handle an additional disk loss, provided there is sufficienttime in between to allow Exanodes to rebuild the data. TheRainX layout requires at least 2 nodes sharing one disk (or more)each.

Simple striping Simple Striping is a disk group data layout, where data is stripedacross the disks, resulting in higher data throughput. Since noredundant information is stored, failure of a disk in the groupcan result in data loss.

iSCSI iSCSI is an IP-based network protocol for accessing SCSI storage.

LUN A LUN (Logical Unit Number) is the identifier of a SCSI logicalunit.

iSCSI Initiator An iSCSI initiator is the endpoint that initiates a iSCSI session.It is the client-side of an iSCSI connection.

Glossary

33

iSCSI Target A iSCSI target is the endpoint that exposes data storage throughLUNs for use by iSCSI initiators.