niif storage services - terena...niif is the hungarian nren, connecting about 450 academic...

Peter Stefan, NIIFstefan@niif.hu

29 June, 2007, Amsterdam, The Netherlands

NIIF Storage Services

Collaboration on Storage Services

Objective of our storage development Brief NIIF introduction Driving force of storage development: users need it! Storage building blocks:

FC, AoE, „Design and develop your own” box.

The two storage node solutions at NIIF How can users access storage volumes? Storage node integration Conclusions

Agenda

NIIF – http://www.niif.hu

NIIF is the Hungarian NREN, connecting about450 academic institutions and 600000 users.

Our service portfolio is as follows: backbone data network (HBONE), videoconferencing services, web, mail, AAI, FTP, DNS services, supercomputers, grids, and data storage facilities.

Financed mostly by the Hungarian Ministry ofEconomics and Transportation.

Who we are?

NIIF – http://www.niif.hu

Who we are?

NIIF – http://www.niif.hu

There is a continuous need for large volume,moderate speed and high availability storagesolutions.

Key usage areas that we know about are: library archives, on-line network monitoring (netflow), videoconferencing, grids, off-site back up of large-volume servers.

Why large storage volumes?

NIIF – http://www.niif.hu

Fibre Channel (FC) SAN/NAS solutions dominatethe market.

Experience with StorageTek (15000 USD/TB), T3,CX500 (6000 USD/TB), SFS (45000 USD/TB).

Generic experience: FC works well high price/{capacity,performance} figures rather large operational costs (disk replacement,

support) not optimal choice for disk-based back ups.

Bulding blocks - FC

NIIF – http://www.niif.hu

Interesting idea: use only Ethernet as storageinterconnection!

ATA-over-Ethernet protocol by Coraid Inc. Key idea: ATA commands are wrapped into

Ethernet frames; the AoE driver strips Ethernetframes, then extracts ATA commands and conveysthem as block devices.

Bulding blocks - AoE

NIIF – http://www.niif.hu

Two conceptually different AoE implementations: Parallel ATA (older, 5 MB/s), Serial ATA (newer, 60 MB/s).

Generic experience with the SATA boxes: moderate performance figures in certain cases very good price/performance value (1200

USD/TB), and prices keep dropping low operational costs (standard disks, very

simple architecture) to build a PByte-scale system further

virtualization layer is required

Bulding blocks - AoE

NIIF – http://www.niif.hu

Use a multi-disk PC chassis with Linux! SATA or SAS disks, standard RAID cards, PCI-express bus system, iSCSI and AoE protocols, universal storage box: GE, Infiniband, 10 GE.

Generic experience with ours: modular can be driven up to the hardware boundaries our record is: 700 MB/s using RAID0 and 450

MB/s using RAID10 no vendor support

„Design and develop your own box”

NIIF – http://www.niif.hu

The purpose is to provide a single, large disk volumefor grid users.

Furbrished from 7 PATA boxes, 2 Cisco Catalystswitches, and 2 64-bit servers.

Involves 7x10x400 MBs of disks. Sum: 28 TBs. Two storage processors are in master-slave layout. Volumes are organized in ten vertical RAID6 arrays

concatenated by LVS. This provides boxredundancy.

Available via grid middleware and SCP-only. Weak points: low IO (50-55 MB/s) , lack of

switch-level redundancy , not scalable .

Node solution 1 - PATA storage

NIIF – http://www.niif.hu

Node solution 1 - PATA storage

NIIF – http://www.niif.hu

The purpose is to provide a single, large disk volumethat can be split to smaller, yet scalable units forgeneric usage.

Made up of 4 SATA boxes, 2 Cisco Catalyst GEswitches, and 2 servers.

GE is used as storage interconnection. Involves 4x15x500 GBs of disks. Sum: 30 TBs. Two Linux storage processors with HA. Volumes are organized in 15 vertical RAID5 arrays

concatenated by LVS. RAID + volume management is performed on

storage proeccors.

Node solution 2 - SATA storage

NIIF – http://www.niif.hu

Full box-, switch-, and storage processor-levelredundancy .

Available via iSCSI for local users . Has virtualization capability, i.e. any boxes and any

storage protocols can be used (FC, AoE, iSCSI,Infiniband, Tape) .

Can be used for building up hierarchical storagesystems .

Not-yet FC-equivalent in terms of e.g. cachecoherency .

IO is about 100-120 MB/s.

Node solution 2 - SATA storage

NIIF – http://www.niif.hu

Solution #2 - SATA storage

NIIF – http://www.niif.hu

Solution #2 - SATA storage

NIIF – http://www.niif.hu

Storage nodes can provide block-level, file-level, andapplication-level access to the large disk volumes.

Block-level: AoE, iSCSI (particularly kind to NRENs).

File-level: NFS, SMBFS, Cluster file systems (GFS, Lustre).

Application-level: SSH, SCP, storage-capable grid middleware.

How to give access to users?

NIIF – http://www.niif.hu

Current status: 2 nodes, lot of plans for the future. Create multiple redundant storage nodes at different

parts of our network, i.e. at regional centers, or atlarge customers.

Integrate them at application level, by using storagemanagement software, like SRM, or GridUnderground (GUG) storage management modules.

The target system is a geographically distributedstorage system providing distributed, replicated, yetsafely protected data sets, and standard interfaces.

First inter-node service: distributed and encryptedbackup service.

Storage node integration

NIIF – http://www.niif.hu

During our storage development activities werevealed that building cost-efficient, reasonable-speed and reliable storage nodes is possible basedon cheap building blocks ands free software.

To handle complexity we believe that application-integrated hierarchical storage systems are needed.

NRENs can play integrating role due to theirpositions.

QoS, monitoring and, in general, strict operationaldisciplines are necessary compared to, say, gridsystems.

There is still a lot to develop: management,advanced monitoring, cache coherency.

Conclusions

NIIF – http://www.niif.hu

?stefan@niif.hu

http://www.niif.hu

Thanks & Questions