iRODS performance test and SRB system at KEKYoshimi Iida @ KEK
Building data grids with iRODS
27 May 2008
27/05/2008 Building data grids with iRODS 2
Outline Performance measurement
Transfer test between CC-IN2P3 and KEK Scaling test for ICAT Concurrent test for ICAT Compare with iRODS and SRB
SRB/iRODS system at KEK
27/05/2008 Building data grids with iRODS 3
Transfer between CC-IN2P3 and KEK
1GB data transfer between CC-IN2P3 and KEK Compare with iRODS and bbcp
The network route go through USA The RTT is about 285ms
KEK, JPCC-IN2P3, FR
NY, USA
27/05/2008 Building data grids with iRODS 4
System used iRODS system at CC-IN2P3
ICAT enabled iRODS server on Solaris 10 Thumpers Sun x4500 (AMD processors) Oracle 10g is on a cluster of dedicated machines Linux file resource from local disk system iRODS 0.9
iRODS system at KEK iRODS server on RHEL3
Intel Xeon 3.0GHz ×4 Linux file resource from local disk system iRODS 0.9
27/05/2008 Building data grids with iRODS 5
From KEK to CC-IN2P3
1GB data transfer during 24 hours window size 4MB number of parallel
streams 16
bbcp often fail to connect
iput
0
10
20
30
40
50
Spe
ed [
MB/s
ec]
bbcp
0
10
20
30
40
50
Spe
ed [
MB/s
ec]
27/05/2008 Building data grids with iRODS 6
From CC-IN2P3 to KEK
iput
0
10
20
30
40
Spe
ed [
MB/s
ec] 1GB data transfer
during 12 hours window size 4MB number of paralle
l streams 16
iput is better than bbcp
bbcp
0
10
20
30
40
Spe
ed [
MB/s
ec]
27/05/2008 Building data grids with iRODS 7
Scaling test Data
Using the same directory at client machine 1000 files of 1000 bytes each
Measurement Ingesting directory and listing the collection Performance measured for every directory
operation 1 collection and 1000 files
27/05/2008 Building data grids with iRODS 8
System used ICAT based on Oracle system at CC-IN2P3
ICAT enabled iRODS server on Solaris 10 Thumpers Sun x4500 (AMD processors) Oracle 10g is on a cluster of dedicated machines
iRODS resource and client on SL4 Dual AMD Opteron Processor 848 Linux file resource from local disk system
ICAT based on PostgreSQL system at KEK ICAT enabled iRODS server on RHEL3
Dual Intel Xeon 2.8GHz PostgreSQL 8.2.5 is running on the same machine
iRODS resource and client on RHEL3 Dual Intel Xeon 2.8GHz Linux file resource from local disk system
27/05/2008 Building data grids with iRODS 9
Ingesting up to 1 million filesiput - Oracle
0
20
40
60
80
100
0 200 400 600 800 1000number of files [x 1,000]
tim
e in
sec
onds
ils - Oracle
0
1
2
3
4
5
0 200 400 600 800 1000number of files [x 1,000]
tim
e in
sec
onds
iput- PostgreSQL
0
20
40
60
80
100
0 200 400 600 800 1000number of files [x 1,000]
tme
in s
econ
ds
ils - PostgreSQL
0
1
2
3
4
5
0 200 400 600 800 1000number of files [x 1,000]
tim
e in
sec
onds
Running the other process
27/05/2008 Building data grids with iRODS 10
Nested collection test Data
Registering same directory at client machine 100 files of 100 bytes each
Measurement Ingesting directory and listing the collection Making nested collection every 10 collection Performance measured for every directory
operation 1 collection and 100 files
27/05/2008 Building data grids with iRODS 11
Nested collection testireg - PostgreSQL
010203040506070
0 50 100 150 200number of files [x 1,000]
time
in s
econ
ds
nestflat
iput - Oracle
010203040506070
0 50 100 150 200number of files [x 1,000]
time
in s
econ
ds
nestflat
ils - PostgreSQL
0
1
2
3
4
0 50 100 150 200number of files [x 1,000]
time
in s
econ
ds
nestflat
ils - Oracle
0
2
4
6
8
10
0 50 100 150 200
number of files [x 1,000]
time
in s
econ
dsnestflat
nest-121 : OCI_ERRORcoll-1220: USER_PATH_EXCEEDS_MAX
27/05/2008 Building data grids with iRODS 12
Concurrent test Data
Ingesting same directory at client machine 1000 files of 1000 bytes each
Measurement Runing multi process at the same time
read operation – ils and iget write operation – iput and ireg mixed operation – iput, ireg, ils and iget
Performance measured for every directory operation DB setting
Set the number of maximum connections to 200
27/05/2008 Building data grids with iRODS 13
Concurrent test for reading metadatails - PostgreSQL ICAT
0.1
1
10
100
1000
0 50 100 150 200 250 300
Number of clients
Tim
e in s
econds
iget - PostgreSQL ICAT
1
10
100
1000
10000
0 50 100 150 200 250 300
Number of clients
Tim
e in
sec
onds
ils - Oracle ICAT
0.1
1
10
100
1000
0 50 100 150 200 250 300
Number of clients
Tim
e in s
econds
iget - Oracle ICAT
1
10
100
1000
10000
0 50 100 150 200 250 300
Number of clients
Tim
e in
sec
onds
Error: connectToRhost failed10 clients 200 clients100 clients 300 clients×
27/05/2008 Building data grids with iRODS 14
Concurrent test for writing metadataiput - PostgreSQL ICAT
1
10
100
1000
10000
0 50 100 150 200
Number of clients
Tim
e in
se
con
ds
ireg - PostgreSQL ICAT
1
10
100
1000
10000
0 50 100 150 200
Number of clients
Tim
e in
se
con
ds
ireg - Oracle ICAT
1
10
100
1000
10000
0 50 100 150 200
Number of clients
Tim
e in
se
con
ds
iput – Oracle ICAT Because of limit of inodes, we
cannot put any more files
10 clients 100 clients 200 clients
27/05/2008 Building data grids with iRODS 15
Concurrent tests for mix - PostgreSQLiput
0.1
1
10
100
1000
10000
0 50 100 150 200
Number of clients
Tim
e in s
econds
ils
0.1
1
10
100
1000
0 50 100 150 200
Number of clients
Tim
e in
se
co
nd
s
ireg
0.1
1
10
100
1000
10000
0 50 100 150 200
Number of clients
Tim
e in
se
co
nd
s
iget
0.1
1
10
100
1000
10000
0 50 100 150 200
Number of clients
Tim
e in
se
co
nd
s
10 clients 100 clients 200 clients
27/05/2008 Building data grids with iRODS 16
Concurrent test for mix - Oracle
ireg
1
10
100
1000
0 50 100 150 200
Number of clients
Tim
e in
se
con
ds
ils
1
10
100
1000
10000
0 50 100 150 200
Number of clients
Tim
e in
se
con
ds
iget
1
10
100
1000
10000
0 50 100 150 200
Number of clients
Tim
e in
se
con
ds
iput – Oracle ICAT Because of limit of inodes, we
cannot put any more files
10 clients 100 clients 200 clients
27/05/2008 Building data grids with iRODS 17
Compare with iRODS and SRB For the scaling test
Data Using the same directory at client machine 1000 files of 1000 bytes each
Measurement Ingesting directory and listing the collection Performance measured for every directory operation
For the nested collection test Data
Registering same directory at client machine 100 files of 100 bytes each
Measurement Ingesting directory and listing the collection Making nested collection every 10 collection Performance measured for every directory operation
27/05/2008 Building data grids with iRODS 18
System used iRODS system at KEK
ICAT enabled iRODS server on RHEL3 Dual Intel Xeon 2.8GHz PostgreSQL 8.2.5 is running on the same machine iRODS 1.0
iRODS resource and client on RHEL3 Dual Intel Xeon 2.8GHz Linux file resource from local disk system
SRB system at KEK MCAT enabled SRB server on RHEL3
Dual Intel Xeon 2.8GHz PostgreSQL 8.2.5 is running on the same machine SRB 3.5.0
SRB resource and client on RHEL3 Dual Intel Xeon 2.8GHz Linux file resource from local disk system
27/05/2008 Building data grids with iRODS 19
Scaling test - iRODS and SRBiput
0
10
20
30
40
50
0 200 400 600 800 1000number of files [x 1,000]
tme
in s
econ
ds
Sput
0500
10001500200025003000
0 10 20 30 40 50number of files [x 1,000]
tim
e in
sec
onds
ils
0
0.1
0.2
0.3
0.4
0.5
0 200 400 600 800 1000number of files [x 1,000]
tim
e in
sec
onds
Sls
0
0.2
0.4
0.6
0.8
1
0 10 20 30 40 50number of files [x 1,000]
tim
e in
sec
onds
27/05/2008 Building data grids with iRODS 20
nested collection –iRODS and SRBireg
010203040506070
0 50 100 150 200number of files [x 1,000]
time
in s
econ
ds
nestflat
Sregister
05
1015202530
0 10 20 30 40 50number of files [x 1,000]
tim
e in
sec
onds
nestflat
ils
0
1
2
3
4
0 50 100 150 200
number of files [x 1,000]
time
in s
econ
ds
nestflat
Sls
00.10.20.30.40.50.6
0 10 20 30 40 50number of files [x 1,000]
tim
e in
sec
onds
nestflat
coll-1220: USER_PATH_EXCEEDS_MAX nest-47: Error Problem running command
27/05/2008 Building data grids with iRODS 21
SRB system for Belle at KEK
Both protocols are authorized by GSI
Setting up on GridFTP server grid-mapfile for SRB user SRB configuration file for serve
r and resource Register LCG user DN on MCAT
MelbourneMelbourneMelbourneMelbourne
NCUNCUNCUNCUASGCASGCASGCASGC
KUKUKUKU
NagoyaNagoyaNagoyaNagoya
CYFRONETCYFRONETCYFRONETCYFRONET
KEK-2KEK-2KEK-2KEK-2
KEK-1KEK-1KEK-1KEK-1
SINETSINETSINETSINET
KEK-LANKEK-LANKEK-LANKEK-LAN
GEANT2GEANT2GEANT2GEANT2
APANAPANAPANAPAN
KEK-DMZKEK-DMZKEK-DMZKEK-DMZ
SRB-DSISRB-DSISRB-DSISRB-DSI
KEK FWKEK FW
HSMHSMHSMHSMMCATMCATMCATMCAT
PluggableExtension
SRBSRB client client
SRBSRB server server
3.5PB3.5PB
GridFTPGridFTP
SRBSRB
NFSNFS
Enhanced Enhanced GridFTGridFTP serviceP service Belle NetBelle Net
dedicated insidededicated inside
Belle NetBelle Netdedicated insidededicated inside
LCG user
Computing FarmStill not integration with Grid
LSFLSF
LCG user
Belle analysis user
27/05/2008 Building data grids with iRODS 22
Plan for iRODS system at KEK Data transfer for J-PARC project
Generate huge amount of imaging data at Tokai About 1PB data in a year in total
Store the data at Tokai storage once, then copy to KEK and distribute for collaborators Storage at Tokai are recycled
Bandwidth between 2sites
will be 10Gbps
KEK
Tokai
60km
27/05/2008 Building data grids with iRODS 23
Acknowledgements Special thanks to Jean-Yves Nief from CC-IN
2P3 for his help to setup iRODS system and supporting at CC-IN2P3
Thanks to Adil Hasan from RAL for his help to these tests
27/05/2008 Building data grids with iRODS 24
Back up
27/05/2008 Building data grids with iRODS 25
Bandwidth Lyon-KEK
0
20
40
60
80
100
120
0 10 20 30 40 50 60
Time [min]
Spe
ed [
MByt
es/s
ec] Lyon to KEK
KEK to Lyon
iperf with some options; -w 4M : TCP window size [Bytes] -P 16 : the number of parallel threads -i 5 : periodic bandwidth reports [sec]
27/05/2008 Building data grids with iRODS 26
Summery of iRODS Performance Transfer from KEK to CC-IN2P3 is not stable but ip
ut is better than simple transfer software (bbcp) iRODS can manage 1 million files stability
In case of very nested collection, PostgreSQL ICAT takes long time to register data
Better performance than SRB Oracle ICAT can handle more than 300 clients at the
same time, but it takes time according to increase the number of clients