kmd – national data storage · igrid2005, sept. 27th, 2005 added value • flexible policy...

iGrid2005, Sept. 27th, 2005

KMD – National Data Storagein the PIONIER network

Maciej Brzezniak, Norbert Meyer, Rafał MikołajczakMaciej Stroiński

iGrid2005, Sept. 27th, 2005

Location

iGrid2005, Sept. 27th, 2005

PIONIER - Available infrastructure

Installed fibers

PIONIER nodes

Fibers+pipes (end of 2005)

Fibers+pipes (end of 2006)

ŁÓDŹ

TORUŃ

POZNAŃ

BYDGOSZCZ

OLSZTYN

BIAŁYSTOK

GDAŃSK

KOSZALIN

SZCZECIN

ZIELONAGÓRA

WROCŁAW

CZĘSTOCHOWA

KRAKÓW RZESZÓW

LUBLIN

KIELCE

PUŁAWY

RADOM

KATOWICEGLIWICE

BIELSKO-BIAŁA

OPOLE

GUBIN WARSZAWA

CIESZYN

SIEDLCEŁÓDŹ

TORUŃP OZNAŃ

BYD GOSZCZ O LSZTYN

BIA ŁYSTOK

GDAŃSKKOSZAL INSZCZE CIN

ZIELON AGÓR A

W ROCŁAWCZ ĘSTOCH OWA

KRAK ÓW RZE SZÓW

LU BLINKIELC E

PUŁAW YRADOM

KATOW ICEG LIWICE BIELSK O-BIAŁA

OPOLE

GUBIN W ARSZAW A

CIESZYN

SIE DLCE

10 GE no des10 GE link s

iGrid2005, Sept. 27th, 2005

PIONIER – transmission 4Q2005

GTS 1,6 Gb/s

GDAŃSK

POZNAŃ

ZIELONA GÓRA

KATOWICE

KRAKÓW

LUBLIN

WARSZAWA

BYDGOSZCZ TORUŃ

CZĘSTOCHOWA

BIAŁYSTOK

OLSZTYN

RZESZÓW

Bielsko-Biała

GÉANT 10+10 Gb/s

KOSZALIN

SZCZECIN

WROCŁAW

ŁÓDŹ

KIELCEPUŁAWYOPOLE

RADOM

BASNET 34 Mb/s

CESNET, SANET

GÉANT/TELIA 2x2,5 Gb/s

DFN 10 Gb/s Gorzów

MAN

10 Gb/s(1 lambda)

2 x 10 Gb/s

1 Gb/s

CBDF 10 Gb/s

PIONIER’S FIBERS

iGrid2005, Sept. 27th, 2005

Project Partners

Academic Computing Center CYFRONET AGH, Kraków

Academic Computing Center in Gdańsk (TASK)

Częstochowa University of Technology

Marie Curie-Skłodowska University in Lublin

Poznań Supercomputing and Networking Center (PNSC)

Technical University of Białystok

Technical University of Łódź

Wrocław Supercomputing and Networking Center (WCSS)

iGrid2005, Sept. 27th, 2005

National Data Storage - goals

• Reliable and distributed data storage systems

• national level

• The system includes: hardware infrastructure and

Backup/Archive management system (server and client)

• The hardware infrastructure: hierarchical storage systems,

file servers, application servers

• Network: uses PIONIER communication channels and the

infrastructure of city

iGrid2005, Sept. 27th, 2005

The target infrastructure:4 KMD nodes (thick)

4 application nodes

Embedded in PIONIER

Features:

High security levelConfidentiality and integrity of data(advanced B/A client)

Higher reliabilitySystem and data availabilityFault tolerance (hardware and

software)

Wide service accessNational (PIONIER+city net)Many SAPs

EfficiencyFast backbone networks, broadband

network accessHigh-end systems (tape libr.,RAIDs)

iGrid2005, Sept. 27th, 2005

Added Value• Flexible policy definition

• High level of security and reliability

Geographical replications – many localisationsCoherency of distribued data – KMD management softwareData coding – used in some scenarios (for advanced client configurations)High accessibility – thanks to appropriate hardware redundancy and the fault tolerance features in KMD management software

• User interface integrated with the new system features

Data encryption, data coherencyEncryption – software and/or hardware based (cert. equipments)

before moving the data from the customer into KMDEnsuring data coherency – by MD5, HMAC

High availability – ensured by the management softwareThe target data location can be chosenThe data replica location can be chosen

• Confidentiality of the archived data

The encryption key is known only by the data ownerThe KMD infrastructure is storing ”raw data”

• Accounting of used resources

Equiped with accounting system (average monthly amount of data, ...)

iGrid2005, Sept. 27th, 2005

Architecture

iGrid2005, Sept. 27th, 2005

Example of usage - FTP/SCP clients

Moving data fromclient location to KMD

iGrid2005, Sept. 27th, 2005

Example of usage - FTP/SCP clients

Data movement KMD-> client

Original location

iGrid2005, Sept. 27th, 2005

Example of usage - FTP/SCP clients

Data Replication

Depends on the predefinedmanagement policy

Done off-line

iGrid2005, Sept. 27th, 2005

Example of usage - FTP/SCP clients

Reading replicated datafrom KMD

Data moved to another node

iGrid2005, Sept. 27th, 2005

Supporting industrial standards – TSM (Tivoli) client

Read/Copy data from/to KMD

Features:- without data replications- compatible with industrialstandards- access to any location- load balancing

iGrid2005, Sept. 27th, 2005

Example – advanced B/A client

Features:- DATA REPLICATION ! - based on B/A Tivoli- API interface- compatible with industrialstandards

Read/Copy data from/to KMD

iGrid2005, Sept. 27th, 2005

End users – Video Monitoring System2004 – 70 cameras 2005 – 85 camerasfinally 200

15 TB – now... 50 TB

backup window23 days only!

iGrid2005, Sept. 27th, 2005

POLICEŁÓDŹ CZESTOCHOWA

POLICEGDAŃSK

KMD node

PIONIER

City Network MAN

iGrid2005, Sept. 27th, 2005

Summary

• User point of view

• Reliable + Secure + User friedly

• It must be cheap

• Service Provider

• Scalable + efficient

• ‘Easy ‘ to manage

• Network demands:

• Nowadays: dedicated channels

• Future: bandwidth on demand, qos

» Backbone infrastructure (p2p)

» User access

iGrid2005, Sept. 27th, 2005

Thank YOU !

meyer@man.poznan.pl