the university of the south pacific - pacnog: the pacific ... · – development • qos tagging...
TRANSCRIPT
The University of the South PacificCampus Network Expectations and Challenges
PACNOG 25 - Fiji
WelcomeEdwin SandysIT Services
Agenda
• About USP• LAN & WAN Infrastructure• Systems & Networks Infrastructure• Cloud Services• Challenges• Future
About USP• USP established 1968 to serve 12 member countries.
– Fiji, Samoa, Vanuatu, Solomons, Tonga, Kiribati, Tuvalu, Nauru, Marshalls, Niue, Cooks, Tokelau.
• Unique Regional Scope;– Covering thousands of islands over 33 million square km of ocean– Around 2 million people and hundreds of distinct cultures– 25,000+ students; 5-8% growth p.a. & 1500 Staff
• Connectivity– 26 Active Sites over 12 Countries– Satellite C & Ku Band (16 Sites – 28 Mbps)– Undersea & Terrestrial Fiber (Fiji Sub Sites, Tonga, Marshalls, Vanuatu &
Samoa)– Upstream via Research & Education Network (AARNet)
Connectivity Cont….• Fiji eXchange Point (IXP) Peering (Early 2018)
– Better service access for Students & Staff– Better VPN Access (Work from Home)– Planning Stages: Livestream lectures for Fiji
• Future for IXP – Later Slide!
LAN & WAN Infrastructure
Roads & Bridges
Satellite Platform Revamp• C Band Dishes Installed 1999 - 20 Years Old
– EOL 15 to 20 Years• Regional Dishes
– 7 New Installs: 2 Non Penetrating & 5 Penetrating Mounts– 3 Refurbished plus non penetrating mount dishes
• Fiji Hub– Refurbish current– Build new antenna
• Enhanced IP Satellite System (iDirect)– 32% efficiency gains of current outbound Mbps– Better enhancements (DVBS-2X with ACM)– Enhanced Modem performance (32APSK Modcods)
Fiber Ring Journey• Business Case Approved - 2008
– Fiber Purchased Approved• Trenching Works Completed - 2009• Cabinet Cleanup & Auditing Completed - 2011
– 70+ cabinets– 35 fiber terminals
• Fiber Pulling and Termination Completed - 2013– Documentation Update & As-Builds
• Cabinet Electrical Wiring & Grounding - 2015• Fiber Ring Switches Purchase Approved - 2017• Fiber ring Online & Operational - Oct 2018
Fiber Ring Infrastructure
FJ VSS Core
ICT – CoreVSS Switch 1
Comms – CoreVSS Switch 2
Sup2T Port 1 & 2Te 1/5/4 to Te 2/6/15Te 1/5/5 to Te 2/6/16
CONSOLE
MODE
STAT DUPLX SPEED STACK
SYST ACTV XPS S-PWRUIDCatalyst 3850 12S
1 2 3 4 5 6 7 8 9 10 11 12
SOH Building
CONSOLE
MODE
STAT DUPLX SPEED STACK
SYST ACTV XPS S-PWRUIDCatalyst 3850 12S
1 2 3 4 5 6 7 8 9 10 11 12
Library Building
CONSOLE
MODE
STAT DUPLX SPEED STACK
SYST ACTV XPS S-PWRUIDCatalyst 3850 12S
1 2 3 4 5 6 7 8 9 10 11 12
SMT Building
CONSOLE
MODE
STAT DUPLX SPEED STACK
SYST ACTV XPS S-PWRUIDCatalyst 3850 12S
1 2 3 4 5 6 7 8 9 10 11 12
CELT Building
CONSOLE
MODE
STAT DUPLX SPEED STACK
SYST ACTV XPS S-PWRUIDCatalyst 3850 12S
1 2 3 4 5 6 7 8 9 10 11 12
FSTE SPAS Building
CONSOLE
MODE
STAT DUPLX SPEED STACK
SYST ACTV XPS S-PWRUIDCatalyst 3850 12S
1 2 3 4 5 6 7 8 9 10 11 12
Aus Aid Building
CONSOLE
MODE
STAT DUPLX SPEED STACK
SYST ACTV XPS S-PWRUIDCatalyst 3850 12S
1 2 3 4 5 6 7 8 9 10 11 12
Land Management Building
CONSOLE
MODE
STAT DUPLX SPEED STACK
SYST ACTV XPS S-PWRUIDCatalyst 3850 12S
1 2 3 4 5 6 7 8 9 10 11 12
FBE SSED Building
Te 1/6/4 toTe 1/0/12
Primary Path
Te 2/6/4 toTe 1/0/12
Primary Path
Te 1/0/11 toTe 1/0/11
Secondary Path
Te 1/0/11 toTe 1/0/11
Secondary Path
CONSOLE
MODE
STAT DUPLX SPEED STACK
SYST ACTV XPS S-PWRUIDCatalyst 3850 12S
1 2 3 4 5 6 7 8 9 10 11 12
Statham Campus
CONSOLE
MODE
STAT DUPLX SPEED STACK
SYST ACTV XPS S-PWRUIDCatalyst 3850 12S
1 2 3 4 5 6 7 8 9 10 11 12
MSP CampusTe 2/6/9 to Te 1/0/12
Te 2/6/7 to Te 1/0/12
Fire
Comms Fire Fiber Redirection
Systems & Networks Infrastructure
Layering on equipment!
General Networking• Scale
– 10 / 40 / 100 Gig• Reduced Footprint
– Less racks = Less Power– Smaller rooms– Less cooling
• Wireless Services– Centralized Management– No Cabling No Problems
(Mesh)• 15000 Feet = 4572m
– Over 350 AP’s– 2000+ Connected Users
• Design is Essential– Redundancy & Resiliency– NSRC Engagement
Wireless Dashboard
Design - Wireless
WIRELESS DEVICES
WIRED DEVICES
ALL SERVICES
USERSSSO, Certificates, Policy
Servers
NAC Server, MDM Server, Provisioning Services
Monitoring & Control
Push Services & Policies
Send
Enforce
Management & Monitor
Management & Monitor
Management & Monitor
Use Register & Authenticate
Design – Layered Approach• Reference Point• Endorsed Direction• Easy Equipment Choices• Application Provision
– Business Critical– Faculty / Departmental– IT Services– Development
• QoS Tagging– End to End
AARNet ISP
10 G Fiber10 G Fiber
2 x 40 G Fiber
HA cluster
TFL Fiber SuvaPrimary AARNet Link
TFL Fiber VatuwaqaSecondary AARNet Link
2 x 10 G Fiber
FGT-3200D-Sec
HA Active
FGT-3200D-Pri
HA Passive
10 G Fiber
Data Center Central Core
ISP Peering
NGO Peering
USP Fiber Interconnect
USP Satellite Interconnect
IXP Peering
USP Campus WAN Sites (Trusted)
Federated Peering
USP Affiliated Sites (Semi-Trusted)
ISP Sites (External)Edge BGP Equipment
VPN
BGP
BGP
SS
Catalsyt 9500 Series C9500-48Y4C
RFID
...
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52
C9500-48Y4CSS
Catalsyt 9500 Series C9500-48Y4C
RFID
...
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52
C9500-48Y4C
GPS
1PPS 10MHz
CLASS 1 LASER PRODUCTGE2GE1GE0 GPS
BOOT
1PVDM0
LINK
CON GE3AUX
MGMT ETHERNET
CRIT
MAJ
MINTAT
WR
1
0
GE4 GE5
GPS
HDD
BITS
BITS
Cisco ASR 1002
21
3
ASR1002-X
GPS
1PPS 10MHz
CLASS 1 LASER PRODUCTGE2GE1GE0 GPS
BOOT
1PVDM0
LINK
CON GE3AUX
MGMT ETHERNET
CRIT
MAJ
MINTAT
WR
1
0
GE4 GE5
GPS
HDD
BITS
BITS
Cisco ASR 1002
21
3
ASR1002-X10 G Fiber
ETHERNET
USB
CONSOLE
AUX
RE
ONLINE/OFFLINE
SYS OK
PS1PS 0 MX80-48TFAN
LINK LINK LINK LINK
0 1 2 3O
I
O
I
0/ MIC 0
1/4
1/5
1/2
1/3
2/8
2/9
2/6
2/7
1/0/
00/
1
1/3/
103/
11
USP to AARNet Rack Link
Internal BGP Equipment
Data Center ServersCISCO NEXUS N9K-C92160YC-X
53 5451 5249 501 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
BCN
STS
ENV
N9K-C92160YC-XCISCO NEXUS N9K-C92160YC-X
53 5451 5249 501 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
BCN
STS
ENV
N9K-C92160YC-X2 x 40 G Fiber
QoS
Business Critical
Faculty / Departmental
IT Services
Development
Server Hyper Converge Infrastructure (HCI)
• Technology– Smaller– Faster– Compact
• Reduced Footprint– Less power– Less racks– Smaller rooms– Smaller cooling– Better Electrical
Server Room Cleanup
HCI Server Compute
# Technology CPU (>2.0 Ghz) Memory (TB) Storage
1Dell VX Rail G410 (6 Nodes)
5 Nodes x 14 Cores x 2 = 140With HyperThreading = 280 2.5 65 TB Effective
(All Flash)
2
Cisco HyperFlex HX240c(3 Nodes)
2 Nodes x 20 Cores x 2 = 80With HyperThreading = 160 3 120 TB Effective
(All Flash)
3IBM ThinkAgile HX552x(8 Nodes)
7 Nodes x 20 Cores x 2 = 280With HyperThreading = 560 2.8
480 TB Raw(Hybrid)
250 TB Effective
• HCI Node Failover N+1
Cloud Services
Determine services that should be placed in the Cloud.
Student Gmail• On Premise
– Google Cloud Directory Sync
– Users & Groups• Cloud Security
– Anti-SPAM– AV– IPS
• Students– 15 Gig mailbox per user– Unlimited Cloud Storage– 30,000 active users– 450 Terabytes of Data
Backup
# Service Classification Backup Frequency
Recovery Testing
FrequencyRetention
Period
1 Business Critical (includes development) Every 6 Hours Every 3 Months 7 Years
2 Departmental & Faculty Weekly Every 6 Months 3 Years
3 IT Managed Weekly Every 6 Months 3 Years
4 Development None None n/a
Recovery
• 3rd Backup Repository– Cloud storage services.– Amazon AWS / MS Azure /
Google / USP– Cost vs Features vs Ease
• Service Recovery– Spins services on the cloud– Business disaster options– No need for secondary data
center
Challenges
What keep the cogs turning!
Challenges for Team• Documentation
– Maintaining records (meticulous)• Standard Operating Procedures (SOP)
– Process driven– Flow of events
• Automation & Central Management– 4 Systems Engineers– 5 Network Engineers– 3 Infrastructure Techs
• Infrastructure Patching vs Application Updates– Patching security platforms, networking gear, etc.– Updating application code and open source
development.
• Testing Tools– Fiber & UTP– Wireless
Challenges Cont.…• Construction / Landscaping
– Fiber Cuts & Pit Damage– Redesign of contractor plans
• Technical Expertise– Constant training renewal– Recruiting appropriate personnel– Internal recruitment process– Maintaining market value
• Security– Securing people (Trust but Verify)– Securing infrastructure– Securing services
• Weather – No Control!
Bridging the Gaps• NOC Monitoring
– Interns with supervisory staff.– Weekday operations from 7am to 10pm– Weekend operations from 10am to 6pm– Other Hours: Automated Notifications
• Detailed Provisioning– Network Operations Center (NOC) – iCinga– Bandwidth Validation – Perfsonar– Network Provisioning – Cisco Prime– Systems Provisioning – vCenter & Prism– Security Monitoring – Firewall Analyzer
Future
What's next on the roadmap?
Future• Connectivity
– Stateful IPv6 Re-Deployment (2019) – Partially Complete– Regional Fiber Connects (join Fiji eXchange Point)– Eduroam – Under Technical Trials– Additional Satellite Providers– Extend 10 Gig (Office Uplink) – 30% Complete– Reduce Firewalls
• Cloud– Office 365 & Disaster Recovery
• Infrastructure– Small is better so consolidate– Localise Content to remote sites (caching)
• Technology Trends– Keep track on current technologies– Extensive Automation & API Integration
WAN Optimisation
VOIPSecurity VPN
Regional Designs - Server
Regional Designs - Wireless