NCICB Systems Architecture

Bill Britton

Terrapin Systems

LPG/NCICB Dedicated Support

NCICB Architecture Components

• Facilities

• Architecture Overview

• Storage

• Processing

• Application Servers

• Systems Management

Facilities

Two locations Primary and Mirror Sites

• Primary Location at Helgerman Court in Gaithersburg

• Mirror site at 6116 Executive Blvd in Rockville

Facilities Management

• Power – APC Silicon for conditioning and Battery

backup– Monitoring and management through web

interface

• Air Conditioning– Self contained computer room A/C with

redundant house system at HC– Dedicated computer room A/C in 6116 on

house water

Production Environment Overview

Cbioapp201SunFire 280R

2 CPUs 900 mhz2 GB RAM

Internal Storage 36 Gb

cbiodb2SunFire 280R

2 CPUs 900 mhz4 GB RAM

Internal Storage 36 GbAttached Storage 150 Gb

Oracle 8.1.7. Enterprise Server

Oracle 9i Application Server

Lpgprod101Sun Enterprise 250

2 GB RAMInternal Storage 50 Gb

Apachezope

tomcat

Cbioapp103Sun Enterprise 250

2 GB RAMInternal Storage 50 Gb

RMI

internet

Wu-ftpd

CBPROD

Ethernet

Raid drive Raid drive

NIHNET

NFS Serverlpgfs104

4 GB Memory4x 400 Mhz processors36 Gb Internal Storage

4 TB SAN Attached Storage

COL-ACT-STA-

1 2 3 4 5 6 7 8 9101112HS1 HS2 OK1 OK2 PS

CONSOLE SAN Switch

Storage

Compaq Rack mountWin 2K Server

Server Server

EVS

Sun Fire 680012 900 mhz processors

12 Gb RAM

Cbioapp301Sun Enterprise 250

2 CPUs2 GB RAM

MediaServer

Lizard TechRealServer

Lpgfs4 400 mhz processors

4 GB RAM

Usershell

server

Dev/Test Environment Overview

Dev101 Dev102

Workstation Workstation Workstation

Lpgprot101 Cbioapp103-s Cbiodev103

Prototypeserver

GEDP Staging ServerRMI Staging Server

Ethernet

Firewall

System Integration Test Servers

Ethernet

Helgerman Court

Ncicbora1

Data

SAIC Oracle DevServer

Cbiodb2-dSunfire V880

4 CPUs750 mhz8 Gb RAM

Internal Storage 128 GbAttached Storage 150 Gb

Cbioapp201-dSunFire 280R

2 CPUs 900 mhz2 GB RAM

Internal Storage 36 Gb

Oracle 9i Application ServerCBDEV

DCPSBR/CaDSR

CDEDesigner

PortalDQI

CBTEST

DCPSBR/CaDSRCDE Portal

CTDEV

OracleClinical

OCTRN

OracleClinical

WorkstationWorkstation Workstation

6116 Executive Blvd.

Current Storage

Hardware:

– Sun Enterprise 420R server – 4 400 Mhz processors – 4 Gb RAM– 2 Compaq HBA fibre channel host bus

adapters – 2 Compaq 8-port fibre channel switches – Compaq EMA12000 storage array – Compaq EMA16000 storage array

Software:• Compaq Secure Path • Veritas Volume Manager• Rsync

Raid drive Raid drive

Server

FC Switch FC Switch

Raid drive Raid drive

Server

FC Switch FC Switch

Ethernet

Gaithersburg - Primary Storage Rockville Location - Mirror Storage

Approx. 6 Tb Raw storageAs of February 19, 2003, fs104's usage is as follows:

Total usable storage capacity: 2879 GB Total allocated to filesystems: 2404 GB (84% of total usable capacity) Total space in use: 1764 GB (61% of total usable capacity) Free space: 404 GB (14% of total usable capacity)

Current Storage (cont’d.)

Shortcomings of current storageFailover to mirror server is time consuming

Single point of failure

420R platform cannot scale

Ports on fibre channel switch are utilized

Volume of storage has outgrown existing replication and backup solutions

Storage Upgrade Solution

• Upgrade Goals Increase server Fault ToleranceIncrease server performanceProvide scalability for server and SAN

• Server SolutionSun Fire V-480 Platforms

2 900 MHz processors4 GB RAMscalable to 4 900 MHz processors and 32 GB Ram

• SAN UpgradeUpgrade fibre channel switch to 32 port

• Cluster SolutionVeritas Cluster Server

• Replication SolutionVeritas Volume Replicator

• Backup SolutionVeritas Flash SnapVeritas Flash Backup

Future Storage

Hardware:

Primary Cluster• 2 Sun Fire V-480 Server • 2 Compaq HBA fibre channel host bus adapters in each server• 2 Compaq 32 port fibre channel switches • Compaq EMA12000 storage array • Compaq EMA16000 storage array

Mirror Cluster• 2 Sun Fire 420R Server • 2 Compaq HBA fibre channel host bus adapters in each server• 2 Compaq 32 port fibre channel switches • Compaq EMA12000 storage array • Compaq EMA16000 storage array

Software: Compaq Secure Path Veritas Volume Manager

Veritas Volume ReplicatorVeritas Flash Snap and Flash Backup

Future Storage Diagram

Mirror

NFS Cluster Server

Primary

NFS Cluster Server

Raid drive Raid drive

Server

FC Switch FC Switch

Raid drive Raid drive

Server

FC Switch FC Switch

Ethernet

Gaithersburg Location - Primary Storage

Rockville Location - Mirror Storage

Server

Data Replication Process

Data is replicated from the storage attached to the NFSCluster Server to the mirror site using Veritas Volume

Replicator.

Server

Dedicated Processing PlatformSun Fire 6800 Platform

12 900 Mhz Processors,

scalable to 24 Processors

with current boards

12 Gb Ram, Scalable to 192 Gb

2 GigE interfaces

2 AC power grids

Front Open View(Click on underlined text to view components)

Zoom In

                                                                                                                                                                                  

                                                                                            

NCICB Application Servers

Web application serverSun Enterprise 2502 400 MHz processors2 GB RAM

RMI ServerSun Enterprise 2502 400 MHz processors

2 GB RAM

Oracle 9i Application ServerSun 280R2 900 MHz processors2 GB RAM

GEDP application serverSun Fire 4802 900 MHz processors2 GB RAM

Media ServerSun Enterprise 2502 400 MHz processors2 GB RAM

FTP ServersSun netra T-1400 MHz processors1 GB RAM

EVSComapq DL 380

Cbioapp201SunFire 280R

2 CPUs 900 mhz2 GB RAM

Internal Storage 36 Gb

Oracle 9i Application Server

Lpgprod101Sun Enterprise 250

2 GB RAMInternal Storage 50 Gb

Apachezope

tomcat

Cbioapp103Sun Enterprise 250

2 GB RAMInternal Storage 50 Gb

RMI

internet

Wu-ftpd

Ethernet

NIHNET

Compaq Rack mountWin 2K Server

Server Server

EVS

Cbioapp301Sun Enterprise 250

2 CPUs2 GB RAM

MediaServer

Lizard TechRealServer

NCICB Application Servers (cont’d.)

Shortcomings of current app serversSingle production apache, zope tomcat server

Single point of failure

E250 platform cannot scale

App Servers Upgrade Solution

• Upgrade Goals Increase server Fault ToleranceIncrease server performanceProvide scalability

• Server SolutionSun Fire V-480 Platforms

2 900 MHz processors4 GB RAMscalable to 4 900 MHz processors and

32 GB Ram

• Load Balancer AppliancesF5 Networks Big IP 5000 Redundant configuration

• Application Oriented ArchitectureRedundant ncicb apache, zope, tomcat serversGEDPCancer Models DatabaseProcessing platformMedia serverOracle 9iAS Server

Systems Management

Management PortalSiteScope monitorsSAR/gnuplot system management portalswusage web stats

RMI monitoringLDAP Tools

http://hcadmin1.nci.nih.gov