Anella Científica: Anella Científica:

A virtualized research networkA virtualized research network

Maria Isabel Gandía CarriedoCommunications Department, CESCA

Terena Networking ConferenceUniversidad de Málaga, 11/06/2009

AgendaAgenda

Anella Científica• Introduction• Functionalities• Evolution

The new core of the network• Topology and equipment• Technology: virtualization in the core• The management• During the deployment• After de deployment

Examples of use

AgendaAgenda

Anella Científica• Introduction• Functionalities• Evolution

The new core of the network• Topology and equipment• Technology: virtualization in the core• The management• During the deployment• After de deployment

Examples of use

About CESCA and Anella Científica About CESCA and Anella Científica

CATNIX created in 1999

Commercial Internet

Public consortium Created in 1991 Formed by:

• Generalitat de Catalunya

• Fundació Catalana per a la Recerca i la Innovació

• 9 Catalan universities

• Consejo Superior de Investigaciones Científicas

Anella Científica created in 1993

About CESCA and Anella CientíficaAbout CESCA and Anella Científica

Anella Científica is the high-speed communications network that connects the universities and research centres in Catalonia

(“Scientific Ring”)

CESCA, as the manager of the Regional Research and Education Network (RREN) in Catalonia and as a Local Internet Registry (LIR) has:• Addresses for the connected institutions:

– IPv4: 84.88.0.0/15– IPv6: 2001:40B0::/32

• An Autonomous System (AS):– AS13041

About CESCA and Anella CientíficaAbout CESCA and Anella Científica

Even for enterprise researchers not only RedIRIS members

CESCA and Anella Científica provide services...CESCA and Anella Científica provide services...

For all the universities and research community not only Ethernet

For all Catalan universities and researchers, not only in Barcelona area not everyone has fibre

For special projects and regular traffic special circuits, dedicated fibres, QoS

(Unicast || Multicast) && (IPv4 || IPv6)

The Anella Membership PolicyThe Anella Membership Policy

A

A.1Public and Private

Universities

A.2Public Research

Organizations& LSF

A.3 Research Centres

and Institutes

A.4Hospital Research

Departments

B

B.1Manager of R+D+i

Program with Public Funding

B.2Institutions with

Relevant Digital Contents for the Scientific and Technical Community

B.3Entities Participating

in R+D+i Projects

B.4Entities of Special

Interest

C

C.1Technological and

Scientific Parks

C.2Other Hospital Departments

Own connection

CAR

Guttmann

2-02

UB

UAB

UPC

UPF

UdG

URV

UdL

UOC

URL

UVic

UIC

UAO

UdA

ESMUC

EUSS

INEFC

ASPB

TERMCAT

CIDEM

FCRI

BAdM

BC

UnescoCAT

CConsultiu

CBUC

CDP

Liceu

VINSEUM

CHV

CSPT

A.1 A.2 A.3 A.4 B.1

B.2

CatSalut

XTEC

B.4

FHAG

XarxaTecla

C.2

PRBB

ParcUdG

C.1

Anella Científica: connected institutionsAnella Científica: connected institutions

BSC

CESCA

CIEMAT

CELLS

CSIC

IDIBELL

Puigvert

SantPau

Dexeus

VHebron

FCRB

FUB

IMIM

FIGTP

BGSE

CESC

CETI

EUPMT

RI

Linked

16-02

≤ 8≥ 10≥ 100≥ 1.000

CTTC

CTFC

IRTA

IGC

IEEC

FBM

i2CAT

ICC

ICIQ

ICFO

IdeG

IDESCAT

IEC

PAM

CRG

IFAE

20-01

21-04

01-01

Anella Científica: projectsAnella Científica: projects

PIC participates in LHC (10 Gbps) i2CAT participates in several European projects:

FEDERICA, Phosphorus,… (10 Gbps) UPC-CCABA participates in EuQoS, MUPBED,… (1 Gbps) New card and 10 Gbps for PASITO Two temporal increases of BW for FCRB at 100 Mbps

(DVTS) Liceu transmits the Opera Oberta course

Anella Científica: EvolutionAnella Científica: Evolution

Time period

Name & logo

Points of access

Technology BW (Mbps) Dedicated circuits

1993-1998 ANELLA CIENTÍFICA

6-8 Ethernet over DQDB

10 (34 Mbps core)

No

1998-2001 15-19 IP/ATM 34-155 (622 Mbps core)

Yes

2001-2003 19-37 IP/ATM 34-155 (2,5 Gbps core)

Yes

2003-2008 37-76 Ethernet over WDM

10-10,000 Yes

2008-2011 76-... IP/DWDM 10-10,000

...

Yes

Changes in the networkChanges in the network

Anella Científica had a central node with separated L3 equipment for institutions type A+B and C.

Level 2 and 3 equipment was the same for most of them. There were several L2 projects involved. In 2008, both the network and the equipment had to be

renewed. Two Calls for tenders to be submitted in July 2007:

one for the network and one for the equipment. Survey among our users to know their needs.

The survey for the last version of Anella CientíficaThe survey for the last version of Anella Científica

Goal: plan the evolution of the RREN (Regional Research and Education Network) and prepare the call for tenders.

Survey was sent in December 2006 to 62 institutions.

We asked about:• Connection needs• Evaluation of our services • Evaluation of our dissemination activities • Planning of new deployments

Results of the survey: BW & points of accessResults of the survey: BW & points of access

2006 2008 2011 2013

Points of access 73 68 69 70

500 Mbps 4 11 17 23

100 - 500 Mbps 16 16 17 14

10 - 100 Mbps 32 30 24 22

10 Mbps 21 11 11 11

Aggregated BW (Mbps) 5,242 11,956 21,625 47,740

2006 2008

Local connections 11 12

10,000 Mbps 0 1

1,000 – 10,000 Mbps 4 5

100 – 1,000 Mbps 2 2

100 Mbps 5 4

Aggregated BW(Mbps) 4,613 19,211

Sustained and progressive growth

Regular traffic connections• Numbers of points of access didn’t

increase considerably• Growth with the access of the Health

and Culture Departments

Special projects

Conclusions of the surveyConclusions of the survey

Requirements for the new Anella Científica:• Increase the BW • Asymmetric and flexible model to cover different needs • Adaptable topology, specially for relevant projects

More reliability: redundancy in the core nodes,in the accesses, etc.

Evolution towards a distributed network:• Two points of presence interconnected• L2/L3 equipment redundant

Connections:• Own fibre• Rent fibre to a operator• Bandwidth from an operator

Anella Científica: 2006Anella Científica: 2006

Internet

Operator

10 Gbps 1 Gbps 100 Mbps 34 Mbps 10 Mbps ≤ 8 Mbps

C. Nord

Anella Científica: 2008Anella Científica: 2008

1. Public and private non-profit Universities2. Official Bodies of Research3. Other non-profit Research centres4. Hospital Research centres

1. Official bodies of R+D management2. Relevant Digital contents institutions3. R+D+i participants4. Special interest for R+D institutions

1. Science and technological parks

2. Other hospital units

A B C

C. Nord Telvent

Operator

Internet

AgendaAgenda

Anella Científica• Introduction• Functionalities• Evolution

The new core of the network• Topology and equipment• Technology: virtualization in the core• The management• During the deployment• After de deployment

Examples of use

Previous topologyPrevious topology

C. Nord

Catalyst 6513 Level 2/3

A+B institutions

Juniper M320Level 3 (RedIRIS)

Nortel

Level 2 (RedIRIS)

Telvent

Cisco 7200Level 3

C institutions

Internet

Special projects

Localconnections

Operator

Topology: what we asked forTopology: what we asked for

Annexus Telvent

Level 3

equipment

Special projects

CESCA-T

Level 2

Level 3

CESCA-CN

Level 2

Level 3

OperatorLocal connections

RedIRIS

Level 2

RedIRIS

Level 3

The equipment we asked forThe equipment we asked for

It had to:• Be modular, scalable, with redundant power supplies, manageable

via SNMP,…• Support BW reservation, QoS, MPLS, multicast, IPv6, SPAN ports• Have good performance• Permit 802.1q VLAN, jumbo and baby giant frames, VLAN stacking• Have filtering mechanisms like access-lists

Virtualization was an improvement

The equipment we asked forThe equipment we asked for

The old equipment could be re-used, improved, its cards could be replaced… It could be used in any of the nodes, for level 2, level 3 or both.

We explained all its characteristics (in fact, the same we were asking for!)…

…Maybe that was the reason why in the 3 offers we had, the equipment we were offered was from the same vendor, with different configurations, but always with two new equipment and reusing the previous switch/router.

We chose the more advantageous: separated L2 and L3 equipment in one node, same equipment in the other node.

Final topologyFinal topology

C. Nord Telvent

Special projects

OperatorLocal connections

Catalyst 6509

Level 3

Catalyst 6513

Level 2

Juniper M320

Level 3 (RedIRIS)

Nortel

Level 2 (RedIRIS)

Catalyst 6509

Level 2/3

DWDM10 Gbps

Giganet2 x1 Gbps

The new core networkThe new core network

Core with two nodes linked with 10 Gbps DWDM + 2Gbps CESCA-CN

• Segmentation of functionalities level 2 and 3:• Catalyst 6513 Sup 720-3B (L2)• Catalyst 6509 Sup720-3BXL (L3)

CESCA-T• Catalyst 6509 Sup720-3BXL (L2/L3)

Use of Virtual Routers Easy creation of new virtual routers (projects,

management, pre-production, …) Separation of routes Layer 3 Balancing/backup More flexibility, redundancy and reliability

Why virtualization?Why virtualization?

It permitted to have many different routers in each node for: • Different types of institutions (A, B and C).• Pre-production environment.• Special projects.• An institution with many points of access that needed a separated

router for its Campus LAN.

It allowed us to take an ISP approach more than a Campus approach.

We had the backup control on the L3 network. Some parts of the L2 network are under the control of the Operator.

Having more nodes would be easier.

ChangesChanges

C. Nord TelventC. Nord

Virtualized core networkVirtualized core network

CoreNetworkCESCA-CN CESCA-T

@REDIRIS CATNIX

@CESCA A, B INET-ALPI@CESCA C

@REDIRIS CATNIX

@CESCA A, BINET-ALPI@CESCA C

MPLS

OSPF

Providers

Institutions

REDIRIS BCN REDIRIS VALORANGE BCN1 ORANGE BCN2

ISP AISP B

ISP C

Core network

CESCA-CN CESCA-T

@REDIRIS CATNIX

@CESCA, A, BINET-ALPI@CESCA, C

@REDIRIS CATNIX

@CESCA, A, BINET-ALPI@CESCA, C

@CESCA, A, BINET-ALPI@CESCA, C@REDIRIS

Logical architectureLogical architecture

Routing IPv4 (example)Routing IPv4 (example)

Redundancy• Between nodes of CESCA• with RedIRIS (or Orange)• at CATNIX

eBGP Anella RedIRIS-VAL

eBGP Anella RedIRIS-CAT

eBGP Anella CATNIX

eBGP Anella CATNIX

BGP RedIRIS CATNIX

CESCA-CN CESCA-T

RedIRIS Barcelona

RedIRIS

iBGP institutions

OSPF infrastructure

RedIRIS València

Institution

eBGP CESCA-CN Institution

eBGP CESCA-T Institution

The management (I)The management (I)

It is more complex• More equipment involved• More dynamic routing (with providers and institutions)• More interfaces to connect with the operator (from 2 to 20)• More VLAN (from 150 to 270)

New syntax, remember not to forget VRF! Some “false friends”…the legacy commands…

• show ip bgp summary (there are no neighbours!) -> show ip bgp vpnv4 vrf <NAME> summary

• ping 10.1.1.1 (oops! It doesn’t work) -> ping vrf <NAME> 10.1.1.1

The management (II)The management (II)

The real friends: • Aliases!

alias exec bgp-ri-adv sh bgp vpnv4 unicast vrf <NAME> neighbors 10.1.1.1 advertised-routes

alias exec pir ping vrf <NAME1>

alias exec pia ping vrf <NAME2>

alias exec par ping vrf <NAME3>

• Includes, but do not abuse…show vlan | inc <INSTITUTION>

show int desc | inc IP6

…

show ip bgp vpnv4 all | inc 10.1.1.0 (No!!)

• Work with a text file, not with the configuration directly

During the deployment (I)During the deployment (I)

In the Catalyst 6500, by default, all the interfaces have the same MAC address!!!cs-6506-24a#show interfaces | include line | address Vlan1 is down, line protocol is down Hardware is Cat6k RP Virtual Ethernet,

address is 00d0.bcf1.ee5c (bia 00d0.bcf1.ee5c) Internet address is 14.18.2.182/16

Vlan2 is down, line protocol is down Hardware is Cat6k RP Virtual Ethernet, address is 00d0.bcf1.ee5c (bia 00d0.bcf1.ee5c)

If the same equipment has more than one interface on the same VLAN (for instance, one with IPv4 and one with IPv6) there are random flaps, packet loss, etc.

During the deployment (II)During the deployment (II)

Address-family ipv4 multicast is not supported in the VRF:

router(config)#router bgp 13041

router(config-router)#address-family ipv4 multicast vrf TEST ^

% Invalid input detected at '^' marker.

Static routing in only one VRF for multicast.

VRF IPv6 are not recommended:

router(config-router)# vrf upgrade-cli multi-af-mode ...Experimental command, not supported by Cisco.

We keep IPv6 in the physical router rather than in a VRF.

Other opened issuesOther opened issues

Tagging of the routes per-VRF and not per-prefix. Supported by Cisco, but not much documentation:• mpls label mode vrf <vrf> protocol bgp-vpnv4 per-vrf

It would be desirable to “enter the configuration” of each VRF instead of writing the VRF each time you write a command.

The name of the VRF can not be changed!!

After the deploymentAfter the deployment

Two bugs affect the IOS causing a memory leak and increasing the CPU (workaround: avoid “show run” commands ).

Maximum number of routes is approximately 106.If the routes increase considerably, we may face a problem. You can not have so many full-routings.

AgendaAgenda

Anella Científica• Introduction• Functionalities• Evolution

The new core of the network• Topology and equipment• Technology: virtualization in the core• The management• During the deployment• After de deployment

Examples of use

Case study: UABCase study: UAB

VLAN from Bellaterra to CESCA-CN for multicast

UAB–CS

Giganet 100 Mbps

FIGTP

Ethernet 20 Mbps

HUVH (unitat docent)

Giganet 500 Mbps

VLAN from each point of access to CESCA-CN & CESCA-T (internet)

VLAN each point of access to Bellaterra forn internal routing

UAB-IGOP

ADSL 4 Mbps

FUB

ADSL 4 Mbps

PRBB (unitat docent)

Giganet 100 Mbps

CESC

Giganet 100 Mbps

UAB (Bellaterra)

Giganet 1 Gbps

HSP (unitat docent)

Giganet 500 Mbps

HSP (Casa Convalescència)

Giganet 500 Mbps

C. Nord Telvent

VLAN with i2CAT for the Electronic Learning Singing project

i2CAT

VLAN PIC for the LHC project

UAB (IFAE/PIC)

DWDM 10 Gbps

Case study: IRTACase study: IRTA

Intra-institution connections

Internet access

IRTA

INET-ALPI@REDIRIS

IRTA has 9 points of access with different technologies, from ADSL to FastEthernet

The point of access in Barcelona makes NAT and filters all the connections, and all of them go to the internet through it

IRTA-MV

IRTA-CT

IRTA-CA

IRTA-CM

IRTA-MO

IRTA-CO

IRTA-SC

IRTA-VA

IRTACESCA

Thanks for your attention!Thanks for your attention!Questions? Suggestions?Questions? Suggestions?

igandia@cesca.es