1 of 5

ENHANCING NETWORK SURVIVABILITY WITH OUT OF BAND

Joseph P. Brenkosh, Edward L. Witzke, Brian R. Kellogg, Ronald R. Olsberg

Sandia National Laboratories1

Albuquerque, NM 87185{jpbrenk, elwitzk, brkello, rrolsbe}@sandia.gov

1 Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’sNational Nuclear Security Administration under Contract DE-AC04-94AL85000.

ABSTRACT

Out of Band Networks used for management and controlare not new. However, their use can now have significantimpact in the area of Homeland Security. This paperpresents the advantages of using an Out of Band Networkto help ensure network survivability and to decrease theMean Time To Repair of any network. It then presents adesign for an Out of Band Network and gives tips basedon the experience gained with such a network at SandiaNational Laboratories.

INTRODUCTION

Any large, geographically dispersed network can haveservice compromised in a variety of ways. Obviously, ifkey components are physically destroyed the only solutionis to have redundancy. Many types of attacks such asbiological, radiological, or the threat of such an attack,have not really damaged the network, however, they makethe network difficult to operate and manage becausenetworking staff can not safely perform their duties insuch a dangerous environment.

More commonly, large geographically dispersed networkshave remote components such as routers and switches,which may come under attack from the network itself orfail from a variety of non-malicious causes. In manycases, most of the management of these network devices isdone remotely by a network operations center or someother central facility. If the remote site is understaffed,not staffed during non-prime hours, evacuated, etc. thefollowing dilemma arises, “How can you diagnose andrepair a network issue remotely when the same network isrequired to access the devices that need to be repaired?”

An Out of Band Network used for management andcontrol is a flat, non-redundant ancillary network [1] thatsolves the problem in this scenario and can also be used tomake any network more reliable. A properly designed

Out of Band Network, can “extend the reach” of thenetwork operations center to the remote site by providingconsole and/or management port connections to networkdevices. By doing so, they enhance network reliability byreducing the Mean Time To Repair (MTTR).

WHY OUT OF BAND?

Most networks today are managed with in-bandManagement Systems. Consequences of in-bandmanagement include the mixing of user traffic withmanagement traffic, and the inability to manage,configure, or repair a network element remotely during anetwork outage or disruption. During outages or severenetwork disruptions, operations personnel must physicallyvisit various pieces of equipment until the offendingdevice is identified and remedied (disconnected,reconfigured, powered off, replaced, etc.).

An Out of Band Network is typically much less complexthan the network that it manages. The lower complexityimproves its reliability. This in turn, ensures itsavailability to perform network management and controlfunctions for the managed network in times of crisis.

Severe network congestion – from inadvertent sources ordeliberate sources – can be debilitating to a network. Anexperiment gone wrong, misconfigured or faultyequipment, or a denial of service attack can prevent in-band troubleshooting due to the traffic load generated inthe network. An Out of Band Network, separated fromthe production traffic, can be used to locate and isolate theoffending device or network segment.

Network sniffers performing troubleshooting cancompound the problems they are diagnosing, by addingtheir own packets to the network they are sniffing.Placing the output of the sniffers onto an Out of BandNetwork prevents the problem of compounding the

2 of 5

congestion and ensures the sniffers availability in times ofnetwork problems.

Using an Out of Band Network, routine maintenance andnetwork management functions on remote equipment canbe performed from a central facility, such as the networkoperations or command center. Even when remote visitsare necessary to replace components, faults can be locatedand identified from the central facility, allowing thetechnician to take the necessary parts and equipment withhim. This saves a “truck roll” by not having to make onetrip to the remote site to diagnose the problem and anotherto fix it.

A large operation with a central facility and many satellitelocations can effectively use an Out of Band Network tomanage their remote locations, regardless of the trafficload on the production side of their network. This canreduce the number of times network technicians musttravel to the outlying locations. The savings are dramaticwhen the remote locations are distant enough that thetechnicians must board an airplane to reach them.

An Out of Band Network is a valuable asset during timesof evacuation, lockdowns/shelter-in-place, orinaccessibility due to biological or radiologicalcontamination. Network faults still need to be addressedin the face of external events precluding direct access tothe network equipment. In fact, these may be the timeswhen a network is most critically needed to facilitate thecooperation necessary to manage and mitigate theseevents. Knowing this, an enemy may combine an attackon the network with a contamination event.

When some sort of lockdown is occurring or personnelmust “shelter-in-place” because of a possible airbornecontaminate, technicians cannot get to remote sites fornetwork troubleshooting and repair. Using an Out ofBand Network, technicians could isolate and possibly fixthe fault from their sheltered location. If the offendingpiece of equipment cannot be reconfigured to work aroundthe problem, it could be remotely disabled to prevent theproblem from disrupting the rest of the network.

When the network facilities have been evacuated becauseof a possible impending attack or are inaccessible due tobiological or radiological contamination of the area, thenetwork can be kept operational from a remote locationusing an Out of Band Network. A situation may arisethrough attack or failed equipment that congests theproduction network. Having Out of Band Network accessfrom remote locations – whether an alternate commandcenter, a different business location, or the networktechnician’s home – can permit network problems to be

isolated and mitigated without having physical access tothe network equipment.

Having remote access via a separated, Out of BandNetwork can improve technician’s response times andMTTR by allowing network problems that occur duringoff-hours (when the network operations center may onlybe minimally staffed) to be resolved from a technician’shome. This cannot be done using the in-band network ifthe fault or attack involves severe congestion. Having theability to troubleshoot and mitigate problems from homeor other off-site locations, reduces the MTTR byeliminating the time needed to get the network technicianon to the site.

OUT OF BAND NETWORK DESIGN

The tactical and/or business decisions to build an Out ofBand Network have been presented. Once the decision tobuild an Out of Band Network has been made, the nextstep is to design the network.

The design of any network is based on three areas [2]:• Technical goals• Technical constraints• Planned applications

Technical goals for an Out of Band Network must bedetermined first. Although improving network reliability,etc. are technical goals, to achieve those goals, morespecific design goals must be determined. Examples oftechnical goals would be remote console or managementport access to switches, routers, and critical servers.

Once the goals have been clearly determined, they must bebalanced by the technical constraints. For a truly Out ofBand Network, the Out of Band Network and theproduction network should not share any equipment orinfrastructure such as media. However, some remote sitesmay only have a single cable running to a location. Thisis a technical constraint, the effect of which can besomewhat mitigated but not eliminated. Size of thenetwork can also be a technical constraint. A very largenetwork with hundreds or thousands of nodes may requirea huge Out of Band Network. However, many of thebenefits can be had by limiting the design to cover onlycore and distribution nodes and maybe a few key accesslayer devices. Technical qualifications of the maintenancestaff are also a constraint. For example, if the productionnetwork is comprised of gear from a particular networkvendor, using gear from a different vendor, which requiresthe staff to learn and maintain gear that is totally foreignto them, must be taken into consideration.

3 of 5

Planned applications and services are the tools that theOut of Band Network will provide. Unless a productionnetwork is small, an Out of Band Network DNS will makeusing it easier. In many cases, telnet or SSH capability isthe only other service that is required. Authenticationservers such as RADIUS and TACACS+ may also beessential. More exotic Network Management Systems canalso be added if desired. Thin clients and servers may beuseful, especially in classified environments.

The person or team designing the network should firstperform extensive stakeholder interviews to determinewhich features would be desired (and which featureswould be useless!) in an Out of Band Network. Astakeholder would be those people responsible formaintaining both the production and the Out of Bandnetworks, funding managers, etc. The designers shouldalso examine best practices in industry, and how thesewere implemented.

Once the information is gathered, the requests of thestakeholders and their concerns must be distilled into awell-defined set of requirements. These will become thetechnical goals, planned applications and services. Theymust then be balanced by the technical constraints. Fromthis, a design can be developed and presented to thestakeholders for comment.

THE SANDIA NATIONAL LABORATORIES OUTOF BAND NETWORK DESIGN

In an effort to follow best industry practices andmodernize its network architecture, Sandia NationalLaboratories recently designed and implemented an Out ofBand Network. The design and the factors supporting itwill now be presented.

After it was determined that an Out of Band Network wasdesired, the Out of Band team performed over 30stakeholder interviews. Although there was quite adifference in opinion on what the network should provide,the team was able to discern the following:

Technical Goals1. Primarily, the stakeholders wanted remote console

port access.2. The network must be secure.3. Maintainability of the Out of Band Network was

very important.4. The network must be reliable.5. It must be convenient and easy to use.6. The design must be scalable to handle future

network growth.

Technical Constraints1. Sandia has a huge network, over 1000 network

nodes. It would not be cost effective to connecteverything.

2. Sandia has many small remote locations with onlyone fiber or cable going to them.

3. The networking staff has high level of skill inCisco Systems equipment, therefore Out of BandNetwork gear which uses or requires Windows,Linux/Unix or other unfamiliar commandsyntaxes would not be the most effective use oftime for the already overworked production staff.

Planned Applications and Services1. Because the stakeholders placed a huge emphasis

on security, SSH version 2 was preferred becauseit encrypts the data stream. Telnet transmits andreceives in clear text.

2. DNS was also required to make the network easyto use.

3. TACACS+ authentication was important so staffcould authenticate in the same fashion they donormally.

4. All of the applications that the networking staffuse regularly such as CiscoWorks, Thin Clients,etc.

It should be apparent that Item 3 of Technical Goals is notcompatible with Item 4 of Planned Applications andServices. However, there was a surprisingly simplesolution to this dilemma. The solution is to use theproduction networks to provide these services to the Outof Band Networks. This is because in many cases thewhole production network is not affected, only part of it.Therefore the services are still available for use. For theoccasions when the whole network is affected, dedicatedOut of Band machines on the Out of Band Network willprovide console access. By using this design solution, thenetwork provides full user services and is also easy tomaintain.

Figure 1 presents the design of the Out of Band Networkused at Sandia National Laboratories. The actual networkis much larger than shown. The key devices andconfigurations will now be discussed.

The Out of Band (OOB) Core Switch/Router providesconnectivity to the Out of Band Network from theproduction network, supports dedicated user access, andconnects the remote devices. Access to the Out of BandNetwork is restricted using Access Control Lists on thisrouter. The Out of Band Network, itself, is flat and

4 of 5

non-routed. In order to get a large enough IP addressspace, four Class C subnets have been combined usingClassless Interdomain Routing. This yields (4 * 256) – 2or 1022 usable addresses. Because of the geographicallydispersed setting of Sandia National Laboratories, theswitch/router has GBIC slots, which are populated withsingle mode GBICs that have sufficient optical power toreach buildings up to 10 kilometers in distance.

Building Y Building Z

OOB Production

Production Network

OOBSwitch OOB Switch

OOB CoreSwitch/Router OOB Mgt.

OOB Dedicated

RS232

100BASE-X

1000 BASE-X

Access Server

Access Server

Managed Devices Managed Devices

OOB Dual-Homed

Building X

Access Server

OOB Switch

Managed Devices

Figure 1. The Out of Band Network at Sandia NationalLaboratories.

The Out of Band Switch in each building performs severalfunctions. It connects the access server back to the core. Italso allows the network to scale to any size, simply byadding more access servers. The switch also enablesdevices with Ethernet management ports to connect to thenetwork.

The Access Server is a device that has both an Ethernetport and RS232 serial ports. These serial ports areconnected to the console port of the Managed Devices(switches, routers, etc.). An import feature of the AccessServer is the concept of IP aliasing. This allows an RS232serial port to be assigned an IP address.

Access to the Out of Band Network is via three methods.Dedicated machines connect directly, and are used whenthe production network is currently unusable. Some key

personnel have dual-homed machines. These machineshave two NICs, one of which is connected directly to theOut of Band Network. Out of Band access for othernetwork maintenance personnel is also possible from theproduction network. As mentioned earlier, this access islimited to approved subnets or hosts.

The last device is the Out of Band Management machine.This machine is used by the Out of Band administrators tosave configurations, perform network testing, and othermanagement functions.

OUT OF BAND NETWORK TIPSThe following are tips that can make building an effectiveOut of Band Network easier:

• Seek stakeholder or customer input early.• Perform site surveys early. Make sure you have

enough space, power, and connectivity.• Keep your design as simple as possible. More

services and functionality equates to more thingsthat can go wrong.

• Media converters or transceivers can lower thereliability of an otherwise robust design.

• If possible, build and test your complete Out ofBand Network in one location first. It makesdebugging a lot quicker and easier.

• There is no best design. Tailor your design to fityour customer. For example, if your customer hasmuch Cisco expertise, consider a Cisco basedsolution. If your customer has Linux expertise,consider a Linux-based solution.

• A good naming scheme is adding “-con” to theexisting device names when adding console portnames to the DNS. For example, if you wantedto access a router named Zeus, you wouldconnect to Zeus-con. This makes the Out ofBand Network easy to use.

• Design with scalability in mind. Once personnelsee the value of an Out of Band Network, theywill want to connect to other devices.

• If you have the opportunity to use alternate routesto remote locations, use them.

• Make security a very high priority. If an intrudergains access to your Out of Band Network,serious consequences can occur.

CONCLUSIONS AND FUTURE DIRECTIONS

This paper presents the reasons why Out of BandNetworks can be an effective tool in the area of HomelandSecurity. It presents design considerations, tips, and alsoan example of a design that was used at Sandia NationalLaboratories.

5 of 5

The design presented is certainly not the only designpossible. Future extensions and enhancements mightinclude adding remote power cycling capability throughpower control devices that connect to RS232 serial ports.Also, examining secure dialup access, which can provideanother means of reaching remote devices, should bestrongly considered. Secure wireless solutions also needto be explored.

Out of Band Networks used for management and controlare not new. They have been used to increase networkreliability and reduce downtime. However, their use cannow have significant impact in the area of HomelandSecurity. By providing Out of Band access, networkdevices under attack or having other problems, can beisolated by disabling interfaces before these problemsspread throughout the network, even if hazardousconditions (biological, radiological, etc.) prohibit physicalaccess to these devices. Network devices connected to anOut of Band Network can also be remotely reconfiguredto adapt to changing events, rebooted, powered off, oreven sanitized in the event that a location has beenoverrun and is now under enemy control.

BIBLIOGRAPHY

1. How Cisco IT-LAN-SJ Achieved HighAvailability, Cisco Systems, San Jose, CA, 2003.

2. Teare, Diane, CCDA Self-Study: Designing forCisco Internetwork Solutions, Cisco Press,Indianapolis, IN, 2004.