Ethernet OAM in Cesnet Backbone

KAREL SLAVICEKCESNET

Botanicka 68a, 60200 BrnoCzech Republic

Abstract: CESNET is an association of universities and Academy of science of Czech republic. It is operatinga national research and education network (NREN) interconnecting all university cities in the republic. SinceATM and SDH technologies were replaced by ethernet technology in WAN data networks CESNET maintains itsbackbone network mostly on dark fibre leased from various vendors. On some lines it it very reasonable to useexternal mediaconverters with special SFP modules. It allows us to employ ethernet OAM technology for bettermonitoring of these lines.

Key–Words: dark fibre, OAM, NREN

1 Introduction

CESNET is an association originated by universitiesand Academy of science of Czech republic in 1996.Its main goal is to operate and develop academic back-bone network of Czech republic. This network startedon lines with bandwidth in hundreds of kbps and stepby step interconnected all academic cities in republic.Contemporary generation of this network is namedCESNET2 and offers bandwidth in order of gigabitsper second on backbone lines.

Since beginning of its existence CESNET offersto its members Internet connectivity and connectivityto scientific networks in Europe. In 1996 CESNEThas build a high speed intercity infrastructure of aca-demic and nonprofit institutions connected to Euro-pean backbone TEN-34 which has interconnected Eu-ropean NREN networks on speed of 34 Mbps.

CESNET is a member of European research or-ganisations like GEANT and is participating in a lotof research project in computer networks.

2 CESNET

In the beginning of the Internet in former Czechoslo-vakia the national backbone network was operated byCzech University of Technology in Prague. It wassince 1991. From the beginning it was clear that all re-search and education institutions like universities andAcademy of science will need to have access to in-ternational computer networks and that they have toparticipate on both funding and maintaining of com-mon networking infrastructure which will offer datanetwork services to all these institutions.

Since 1996 CESNET was solving project of czechministry of education named TEN-34 CZ. Goal of thisproject was to construct national backbone networkwith technical parameters corresponding to transeuro-pean research backbone named TEN-34. (34 is thenominal bandwidth used in backbone lines.) Back-bone of TEN-34 CZ (how the network was named atthat time) was build on ATM technology. Separatenodes was equipped with ATM switches and routersand interconnection of these nodes was done via ATMswitches. In 1997 migration of CESNET membersnetworks to ATM backbone was finished. At this timestarted discussions about possibilities for further de-velopment of network especially with respect to newtransport protocols and new applications like ip mul-ticast, IPv6, packet voice services and others.

3 History of Czech NRENAt the beginning of networking in former Czechoslo-vakia the network topology was rather poor. It wasa tree structure with root in Prague. From Praguewent IP line to Linz as a first international connectiv-ity. Large cities like Brno, Ostrava, Pilsen or HradecKralove was connected to Prague via leased lines withtypical bandwidth 128 - 512 kbps. Smaller cities wasconnected via leased lines to the ”second level” nodesmentioned above. Typical bandwidth of such connec-tion was 19.2 kbps.

In the next step main lines were upgraded to 1 -2 Mbps and mostly all university cities was connectedto Prague. At the same time the topology of backbonewas a little bit improved. From tree structure we havemoved to some kind of ring topology. At least con-

Proceedings of the 10th WSEAS International Conference on COMMUNICATIONS, Vouliagmeni, Athens, Greece, July 10-12, 2006 (pp286-290)

Figure 1: Physical Dark Fibre Topology of CESNET Backbone

nectivity for largest cities was done not only to Praguebut also to another suitable city of similar size. Con-nectivity to smaller cities was upgraded to 64 - 256kbps. At the beginning of Internet in our republic thewas no other ISP than academic community so firstcommercial users of Internet had no other possibil-ity than to use CESNET as an ISP. On the other sideCESNET approach was to deploy Internet to publicfor education purposes and use revenue from Internetcustomers for improvement of backbone and deploy-ment of Internet to more cities.

Since 1996 when TEN-34 CZ project started thebackbone of NREN started using ATM technology.At that time the ATM backbone was a tree structurebased on microwave E3 lines. For price optimisationsome ATM switches was housed at E3 lines providerlocations. Backup connectivity was solved on legacytechnologies i.e. leased lines with bandwidth 2 Mbps.In 1997 migration of CESNET members networks toATM backbone was finished. As a next step was up-grade of main backbone lines to STM-1 speed. Thisstep didn’t provide more benefit than bandwidth im-provement.

In 2000 LANE protocol in backbone network wasreplaced by MPLS on ATM transport layer. The maingoal of backbone network at that time was to pro-vide reliable and stable IP connectivity to its mem-bers. Next to his main goal are in terms of researchplan started experimental and supplemental services.In this year was build first line based on dark fibre. Itwas line Prague - Brno and POS STM-16 was used

on this line. The line is about 300km in length and 3regenerators was placed in proper locations along theline. In 2001 was ATM step by step replaced by POSand gigabit ethernet (GE). It was run on a mix envi-ronment of dark fibre and leased line POS STM-16.

A next step was movement to complete dark fi-bre infrastructure. The reasons for this step are thefollowing: at speed of GE and STM-16 dark fibre ismore economical than leased lines and bandwidth im-provement and technology migration. More over thebackbone moved to NIL (Nothing In Line) technologyon mostly all lines. By usage of fibre optical ampli-fiers it is no more necessary to maintain any regener-ators somewhere in line. The maintenance should bedone only in network nodes which are located at uni-versities so that problems with transport of staff andmaterial to remote locations somewhere in field dis-appears.

4 Dark Fibre in CESNET BackboneDark fibre especially with combination with NIL tech-nology provides a lot of advantages. Wen can freelyuse whichever networking technology we want. CES-NET is using this technology on all backbone lines ex-cept lines with DWDM technology. The reason is sim-ple: DWDM is used to provide data channels goingthrough several backbone nodes. The real length ofdata channel may be several hundreds of kilometres.For this reason we can’t use optical amplifiers withtoo high amplification because the higher amplifica-

Proceedings of the 10th WSEAS International Conference on COMMUNICATIONS, Vouliagmeni, Athens, Greece, July 10-12, 2006 (pp286-290)

tion the bigger signal distortion. Of course DWDM isused on dark fiber lines as well. The schema of ”L1”topology i.e. dark fibre infrastructure is on picture 1.

CESNET is leasing dark fibre lines from manyvendors. On certain lines is used DWDM technologybought as a complete solution including in-line opticalamplification. On lines without DWDM we use opti-cal amplifiers on data termination points only. Firstreal usage of this principle in CESNET backbone isrun on line Prague - Pardubice. This line is 189kmin length and is equipped with EDFA (Erbium DopedFibre Amplifier) since 2001. EDFAs on this line areused as boosters i.e. are connected below transmitteron both sides of the line. The longest line equippedwith this technology is line Brno - Ostrava. The lengthof this line is about 235 km. This line is equipped with4 EDFAs: output of each transmitter is connected toEDFA booster. Output signal of the booster is about+22dBm. Before each receiver is connected EDFApreamplifier. This line is now used for transport ofGigabit ethernet. With similar setup it was used forPOS STM-16 type of traffic. It was replaced by giga-bit ethernet due to cost of data termination equipment.

Lines up to 110 km usually have attenuation be-low 30dB. On such lines is transported unamplifiedgigabit ethernet. For this type of lines the CWDMGBICs or SFPs are used. The reason is they haveabout 2dBm better power budget than traditional ZXgigabit ethernet pluggables.

Lines to some smaller locations like Karvina orJindrichuv Hradec are constructed as a single fibrelines. The reason for usage of this type of lines ispartially price of leased dark fibre lines and partiallypure dark fibre lines availability in some areas.

5 Cost-effective solution for midsizedlines

Today smart SFP modules with power budget of 36dBare available on the market. Such SFPs can be usedon lines with length about 160km. However main net-working equipment vendors used to limit spectrum ofgigabit ethernet pluggables (GBICs or SFPs) usable intheir routers and switches to a relatively small num-ber of approved types. Till now it prevented us forutilising of a lot of interesting SFP modules with highpower budget or another special properties suitable forutilisation in CESNET backbone network.

For utilisation of new models of SFPs which arenot yet tested by our networking equipment manufac-turer we propose to use small external equipment serv-ing as a ”transponder”. The idea is easy to understandfrom the picture 2.

There are some basic requirements for ”transpon-der” equipment:

• redundancy of power supply and central proces-sor unit if any is necessary

• reasonable level of modularity• manageability via SNMP

For many possible applications of this technology inCESNET network we appreciate ethernet OAM asspecified by IEEE 802.3ah. For this reason we haveconcentrated on equipment supporting this protocol.We have tested SFP-to-SFP media converter MRVmodel EM316-GRMAHSH. Now we are performingproduction testing of this equipment on line Brno-Zlinwhich has about 130km in length and 33dB attenua-tion.

SX

SFP-X

MCSX

Router

SX

Router

SX

SFP-X

MC

Figure 2: Usage of external ”transponder” for utilisa-tion of special SFP modules

As SFP modules were used two types of SFPsfrom Optoway Ltd.:

• Single fibre modules SPB-7780-1510 resp SPB-7780-1590

• Legacy (2-fibre) modules SPS-73160The first one has power budget 25dB and is suitablefor most of single fibre lines used in Cesnet backbone.The second on has power budged 36dB and now istested on the line Brno-Zlin.

For testing purposes we have utilised vendor spe-cific management application for monitoring and con-figuration of transponders. This application workedfine but is not easy to integrate it into existing net-work management system. For this reason we haveprepared a perl utility which communicates with MRVmediaconverters via SNMP and provides suitable in-terface for integration into network management sys-tem used for Cesnet backbone.

6 Ethernet OAMEthernet OAM (Operations, Administration andMaintenance) was originally introduced by EFMA

Proceedings of the 10th WSEAS International Conference on COMMUNICATIONS, Vouliagmeni, Athens, Greece, July 10-12, 2006 (pp286-290)

(Ethernet in the First Mile Alliance) and later stan-dardised by IEEE in 802.3ah specification. The mainpurpose of this protocol was to overcome some ineffi-ciency or weakness of traditional enterprise manage-ment protocol i.e. SNMP. Although SNMP provides avery flexible management solution sometimes it is in-adequate to management task. First of all it assumesoperational underlying network because it relays onIP connectivity. However we need management func-tionality even if (and first of all if) the underlying net-work is not operational.

Carriers looked for management capabilities atevery network layer. The ethernet traditionallydoesn’t offer management capabilities so the 802.3ahis mostly the only way for management of this layer.Though it was originally intended for EFM (ethernetin the First Mile) applications it is very suitable forethernet base backbone lines.

Octets Field Fixed Value

6 DST Address 01-80-c2-00-00-026 SRC Address2 LengthType 88-091 Subtype 032 Flags1 Code

42-1494 Data + Padding4 FCS

Figure 3: The OAMPDU structure

OAM information is conveyed in slow protocolframes called OAM Protocol Data Units (OAMPDU).OAMPDUs traverse a single link only, MAC clientslike bridges and switches do not forward them. Theoverview of OAMPDU structure is on figure 3. Thereare six types of OAMPDUs distinguished in the Codefiled of OAMPDU:

• Information OAMPDUs• Event Notification OAMPDUs• Variable Request OAMPDUs• Variable Response OAMPDUs• Loopback Control OAMPDUs• Organization Specific OAMPDUs

Once link comes up on the interface the OAM dis-covery begins. It allows local DTE to detect OAMon the remote side. Once OAM support is detectedboth ends of the link exchange state and configurationinformation e.g. mode, PDU size, loopback support

etc. and OAM is enabled on the link. After loss oflink signal or not hearing OAMPDUs for more than5s the discovery process is restarted. The OAM modeis either Active or Passive. Active mode should be onprovider end of the link. Device in this mode initi-ates the OAM discovery, sends Information OAMP-DUs and may send Event Notification, Variable Re-quest and Loopback Control OAMPDUs. Passivemode should be configured on customer end of thelink. Device in this mode waits for remote side to initi-ate OAM discovery process, sends Information OAM-PDUs, may send Event Notification OAMPDUs, re-spond to Variable Request by sending Variable Re-sponse OAMPDUs and react to received LoopbackControl OAMPDUs. This device may not send Vari-able Request and Loopback Control OAMPDUs.

OAMPDUs are send at maximum rate of 10frames per second as defined in Annex 43B of IEEE802.3. Some properties of OAM like remote loopbackcontrol are designed mainly for service providers andare not used in Cesnet environment. We are using onlyVariable Request/Response types of OAMPDUs forstatus monitoring and part of Organization SpecificOAMPDUs for setting up properties of remote me-diaconverter interfaces like speed and duplex on UTPinterface.

7 OAM MIBs

Ethernet OAM is operating on MAC client to MACclient basis. The device running active mode of OAMshould communicate with network management sta-tion and provide status information on both local andremote end. The networking device running OAM inactive mode uses Variable Request OAMPDU to re-trieve status information from the remote side. Theprinciple is illustrated on picture 4.

Network Management Station

Router RouterMediaconverter Mediaconverter

IP SNMP

IP SNMPIP SNMP

Figure 4: Management of remote mediaconverter viaOAM

Structure of variables that can be retrievedVariable Request OAMPDU is defined by IEEE802.3 standard in the Annex 30A. The structure of

Proceedings of the 10th WSEAS International Conference on COMMUNICATIONS, Vouliagmeni, Athens, Greece, July 10-12, 2006 (pp286-290)

this variable list corresponds to SNMP MIB tree.The Variable Request OAMPDU contain one ormore (limited by frame size) Variable Descriptors.Variable descriptor consists of two fields: oneoctet Variable Branch and two octets Variable Leaf.This approach assumes that network managementstation will ask for MIB variables of the formatiso(1).member-body(2).us(840).ieee802dot3(10006).csmacdmgmt(30).branch.leaf . Where branch is oneof

managedObjectClass 3package 4nameBinding 6attribute 7

Most important information are in the attributebranch. Here are items like FramesTransmit-tedOK, FramesReceivedOK, OctetsTransmittedOK,OctetsReceivedOK, etc.

Network management station uses legacy SNMPfor retrieval of information from networking de-vice running OAM. SNMP MIB variables may bemapped into OAM variables which are retrievedfrom remote side of the link via Variable Re-quest/Response OAMPDUs. Not all vendors ofnetworking devices use MIB branch iso.member-body.us.ieee802dot3.csmacdmgmt. The reason is thatsome networking equipment were delivered beforeIEEE and IETF finished corresponding standards.

In the case of MRV a combination of ven-dor specific branch of MIB tree and standard.iso.org.dod.internet.mgmt.mib-2.interfaces.ifTablewith special ifIndex is used. If the ifIndex of local in-terface is N then the ifIndex of corresponding remoteinterface is 5000 + N. This approach is used for alltraditional status information and statistics counterslike ifAdminStatus, ifOperStatus, ifInOctets, ifIn-UcastPkts, ifInNUcastPkts, ifInErrors, ifOutOctets,ifOutUcasPkts, ifOutNUcasPkts, etc. The vendorspecific branch of MIB tree is used for obtainingSFP related information like SFP vendor, serialnumber, in case of SFPs equipped with managementinformation interface yes optical TX and RX power,laser bias etc. All these information can be accessedvia .iso.org.dod.internet.private.enterprises.nbase.nbsCmmc.nbsCmmcPortGrp.nbsCmmcPortTable.

8 Conclusion and future work

Originally external mediaconverters was consideredonly as an economically more suitable solution formid-range fibre optics lines. In the second step prop-erties of ethernet OAM were evaluated and countedas very interesting for application on Cesnet giga-bit ethernet backbone lines. OAM is very useful

tool for monitoring and management of dark fibrelines equipped with gigabit ethernet technology. Be-cause this technology starts to be deployed to mostof today’s dark fibre lines instead of traditional timedivision multiplexing technology (SDH/SONET) itis extremely important to provide to ethernet tech-nology management capabilities comparable withSDH/SONET.

We have successfully tested such kind of manage-ment on one intercity line. As a next step we plan todeploy ethernet OAM technology to most of single-channel backbone lines in Cesnet network. Especiallyon lines which needs some external mediaconversionor signal amplification equipment this technology willbe very useful because it can provide us basic manage-ment information of the topologically nearest remotedevice.

References:

[1] Yearly reports of projects and scientific re-search plan solved at CESNET available atwww.cesnet.cz

[2] IEEE 802.3ah standard[3] MRV inc. Optical Ethernet Services Demarca-

tion.

http://www.mrv.com/library/download.php?ctl=MRV-WP-MRV-AH-Req1&type=pdf300_url

[4] DOSTAL,Otto - FILKA,Miloslav - SLAVICEK,Karel. MPLS in Practise - The New Back-bone of NREN. In TELECOMMUNICATIONSAND SIGNAL PROCESSING - TSP 2001.Brno : UTKO VUT, 2001. ISBN 80-214-1962-8, SEPTEMBER 5 TO 6, 2001, BRNO, CzechRepublic.

[5] DOSTAL,Otto - FILKA,Miloslav - SLAVICEK,Karel. Videoconference using ATM in medicine.In Telecommunications and Signal ProcessingTSP 99. Brno : FEI VUT BRNO, 1999. ISBN80-214-4154-6, Brno.

[6] O. Dostal - M. Javornik - K. Slavicek - M.Petrenko - P.Andres, Development of RegionalCentre for Medical Multimedia Data Processing,Proceedings of the Third IASTED InternationalConferee on Communications, Internet and In-formation Technology. St.Thomas, US Virgin Is-lands: International Association of Science andTechnology for Development- IASTED, 2004.ISBN 0-88986-445-4, p.632-636.

Proceedings of the 10th WSEAS International Conference on COMMUNICATIONS, Vouliagmeni, Athens, Greece, July 10-12, 2006 (pp286-290)