CLUSTER BOOT ISSUESShawn MarriottCOSC 3P93

What is a Cluster?

A cluster is a collection of individual autonomous nodes working in concert to create the illusion of a single system.

The trouble with nodes

If you must consider the state, and configuration of each node in a cluster the illusion of the single system is lost. As the size of a cluster grows the management of individual nodes becomes tedious, and error prone.

A Paradox

How does a node get the information it needs to boot, if it needs to boot to get the information it needs ?

Booting a computer

Part of the startup procedure for most computers is to initialize a bootstrap, which in turn starts the operating system. Where to find the bootstrap is generally configure on a per machine basis in the bios. Common places that are searched for the bootstrap are the hard drive(s), optical drives, solid state devices, and floppies.

Or at least convince the nodes to boot…….

How do you boot a cluster?

A Sub Ideal Solution:

Manually configure every node in the cluster…

An ideal solution:

Unpack the computer. Plug in the computer. Turn on the computer. Done.

How could this be done?

If the nodes in our cluster are computers connected to a network. Then when a computer starts up have it use the network to broadcast information about itself to a server, and have the server provide configuration information and a boot strap.

In the beginning….

There was the Reverse Address Resolution Protocol (RARP). RFC 903, 1984.

Next try……

Bootstrap protocol (BOOTP). RFC 951, 1985

Sample BootP config

node01:ht=ether:ha=080009030166:ip=15.19.8.2:sm=255.255.248.0:gw=15.19.8.1:bf=/bootloader

node02:ht=ether:ha=080009030176:ip=15.19.8.3:sm=255.255.248.0:gw=15.19.8.1:bf=/bootloader

node03:ht=ether:ha=080009030186:ip=15.19.8.4:sm=255.255.248.0:gw=15.19.8.1:bf=/bootloader

node04:ht=ether:ha=080009030196:ip=15.19.8.5:sm=255.255.248.0:gw=15.19.8.1:bf=/bootloader

Another go around

Dynamic Host Configuration Protocol (DHCP).

RFC 1531, 1993.

Sample DHCP Configuration

 subnet 192.168.0.0 netmask 255.255.255.0

{ range 192.168.0.10 192.168.0.49; option subnet-mask 255.255.255.0; option broadcast-address

192.168.0.255; option routers 192.168.0.1;

filename "pxelinux.0"; next-server 192.168.0.100;

}

One more try?

In 1999 Intel releases the Wired for Management Framework (WMF)

The framework was mostly ignored, but two interesting parts of the specification endure.

Preboot Execution Environment (PXE). Boot Integrity Services (BIS).

What is PXE?

PXE is a client designed to work with DHCP and TFTP to retrieve boot strap information, it also defines APIs to allow a loaded boot strap to query a local host for additional information

What is BIS?

BIS enables a PXE client to examine a digitally signed boot image. This provides a mechanisms to verify the integrity of a supplied boot strap image.

How is PXE implemented? If you want to use a network to provide

configuration information for a node, then you better have a network card (NIC).

Since you must have a network card, it seems reasonable to put the PXE client on the network card, and have the bios treat network cards as a bootable device.

Intel’s vision

What is a Network Bootstrap Program(NBP)? NBP is a binary executable file, specific

to a give CPU’s architecture. They are small, usually less than 512KB

in size. What an NBP does is up to whoever

creates it. The PXE specification does not go into any detail on NBP.

PXE Work flow with an NLB

What is involved in setting up a PXE environment?

A Node with a PXE client set as the boot device.

A DHCP server TFTP server

Sample DHCP Configuration

 subnet 192.168.0.0 netmask

255.255.255.0 { range 192.168.0.10 192.168.0.49; option subnet-mask 255.255.255.0; option broadcast-address

192.168.0.255; option routers 192.168.0.1;}

Sample proxy DHCP config

subnet 192.168.0.0 255.255.0.0

{

vendor pxe

{

bootstrapserver 192.168.0.100 # TFTP server ip address.

#Type, SystemArch, MajorVers

pxebootfile 1 2 1 window.one 1 0

pxebootfile 2 2 1 linux.one 2 3

pxebootfile 1 2 1 hello.one 3 4

client 6 10005a8ad14d

{

pxebootfile 1 2 1 aix.one 5 6

pxebootfile 2 2 1 window.one 6 7

}

}

}

What PXE does right

It is a standard feature on modern NICS, and mother boards with integrated NICS

A PXE client knows the architecture of the node it is running on, and can request an appropriate NBP

It builds upon existing technologies (DHCP, TFTP)

It has extensions to authenticate the boot server, and verify the integrity of a downloaded NBP.

The problems with PXE

PXE has limited knowledge of it’s host. PXE relies on DHCP, which is inefficient on

large networks PXE relies on TFTP, which is impractical for

large files, or many concurrent file transfers If you need different NBPs for different

nodes you must uniquely identify each node(By MAC address) and group them accordingly in DHCP or Use a DHCP proxy server and separate the management of network address and images

Questions and Discussion

References

Thomas L. Sterling , 1998. Beowulf Cluster Computing with Linux. Cambridge, Massachusetts: The MIT Press

Intel Corporation, 1998. Boot Integrity Services Application Programming. Version 1.0.ftp://download.intel.com/design/archives/wfm/downloads/bisspec.pdf

Intel Corporation, 1999. Preboot Execution Environment (PXE) Specification. Version 2.1.http://download.intel.com/design/archives/wfm/downloads/pxespec.pdf

http://www.ietf.org/rfc/rfc903.txthttp://www.ietf.org/rfc/rfc951.txthttp://www.ietf.org/rfc/rfc1531.txthttp://www.beowulf.orghttp://en.wikipedia.org/wiki/Preboot_Execution_Environmenthttp://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp