developing an agricultural monitoring system from remote sensing data using gridrpc on ninf-g shamim...
TRANSCRIPT
Developing an Agricultural Monitoring Developing an Agricultural Monitoring System from Remote Sensing Data System from Remote Sensing Data
Using GridRPC on Ninf-GUsing GridRPC on Ninf-G
Shamim Akther, Yann Chemin, Honda KiyoshiAsian Institute of Technology, Thailand
Itthichok Jangjaimon, Putchong UthayopasKasetsart University, ThailandAnd Thai National Grid Project
2
MotivationMotivation
• Recent advances in data assimilation of satellite imagery for the purpose of agricultural monitoring led to the increasing demand for computing power
• There is a need for technology and platform that can meet the requirement
3
SWAP Model and StructureSWAP Model and Structure
Adopted from Van Dam et al. (1997)Drawn by Teerayut Horanont (AIT)
4
SWAP-GASWAP-GA
• Know SWAP Input parameter – sowing date, soil property, Water management, etc
• Finding SWAP Crop model
Comparison of Satellite LAI and Simulated LAI
0
1
2
3
4
5
0 30 60 90 120 150 180 210 240 270 300 330 360
DOY
LA
I
LAI_sat
LAI_sim
5
SWAP Model Parameter identificationSWAP Model Parameter identification - Data Assimilation using RS and GA- Data Assimilation using RS and GA
0.00
1.00
2.00
3.00
4.00
0 45 90 135 180 225 270 315 360
Day Of Year
Eva
po
tra
nsp
iratio
n L
AI
RS ObservationRS Observation
SWAP Crop Growth ModelSWAP Crop Growth Model
SWAP Input ParametersSWAP Input Parameters
sowing date, soil property, sowing date, soil property, Water management, and etc.Water management, and etc.
LAI, LAI, EvapotranspirationEvapotranspiration
0.00
1.00
2.00
3.00
4.00
0 45 90 135 180 225 270 315 360
Day Of Year
E
avpo
tran
spira
tion
LA
I
FittingFitting
LAI, LAI, EvapotranspirationEvapotranspiration
Assimilation by Assimilation by finding Optimized finding Optimized
parametersparameters
By GABy GA
RSRS ModelModel
6
Pixel pool
Result pool
Best fit population
Sequential Version
Sequential VersionSequential Version
• Take satellite image and loop through each pixel
• For each pixel perform GA processing
7
ChallengesChallenges
• The calculation time for identify SWAP parameters only for 1 pixel (1 sq.km) takes several minutes to 30 minutes.
• Thus, a RS image of 1000 x 1000 sq.km of 1000x1000 pixels will take more than 50 years (30min x 1000 x 1000) is not acceptable.
• Solutions– Parallel Computing on
Cluster and Grid
• Longitude: 100.008133
• Latitude: 14.388195
8
Thai Grid InfrastructureThai Grid Infrastructure
Network
SPIRIT
PALM
OPTIMA
ENQUEUE
GASS
AMATA
MAEKA
WARINE
KU
ThaiGrid User
ThaiGrid PortalTgcc.cpe.ku.ac.th
Submit
Grid Job Scheduling
CU
KMUTNB
KMUTT
AIT
SUT
CAMETAINCA
9
Programming ModelProgramming Model
• Message Passing Using MPI– Widely adopted standard– Work efficiently on Cluster– Can work at some level on Grid
• Grid RPC– Remote procedure call programming model
on Grid system– GGF adopted as a standard programming on
Grid– Several Implementation : GridSolve, Ninf-G
10
GridRPC Programming ModelGridRPC Programming Model
• Application partition into client/server– Multiple client can be used to achieve
parallelism using asynchronous call
• Role– Server side
• Implement the algorithm as a remote procedure
• Using Grid for security mechanism and execution management
– Client side • Implement main workflow of the application
• Handling input/output for user
• Call Remote Functions at Server using GridRPC API
11
Ninf-G programming modelNinf-G programming model
• A software package which supports programming and execution of Grid applications using GridRPC
• Advantages– Simple programming model that work
efficiently over lossely copuple heterogenous system
– Hiding most of the grid complexity from programmer and developer
Low level Grid Middleware
Grid RPC
Grid application
Grid Fabric Layer
12
Partitioning strategy on clusters and GridPartitioning strategy on clusters and Grid
• There are 3 possible partitioning strategies– Population Distribution – Pixel Distribution– Hybrid Distribution
• Group compute nodes as virtual group• each virtual group take care one pixel• inside the group, distributing population to members
• Pixel and population distributed has been implemented using MPI and reported in our previous work
13
Population Distribution StrategyPopulation Distribution Strategy
Pixel pool Result pool
Eval Eval Eval
GA
Set of Population
fitness
14
Pixel Distribution StrategyPixel Distribution Strategy
Pixel pool Result pool
GA GA GA
Image
Pixel
Results
15
Hybrid Distribution StrategyHybrid Distribution Strategy
GA
Pixel pool Result poolImage
Pixel
Results
EvalEval Eval
population
Cluster
GA
Pixel
Results
EvalEval Eval
population
Cluster
16
Extension to the GridExtension to the Grid
• Modify and adjust code to change distributing middleware from MPI to Ninf-G
• Implement secure data-transferring method for Grid instead of using NFS for cluster environment
• Gain benefit of Grid computing– More effective– More scalable– More security
17
ArchitectureArchitecture
18
Table 4: Server Program Model with Ninf_G //GIS program extract pixels from the Modis RS image and //pixel information will be written in separate file for each pixel. for pix_no = first to last { //Grid Remote Procedure Call environment initialization. grpc_initialize(config_file); for(pixel is 1 to 16) //let one time we manipulate 16 pixels { //Each pixel create one handle grpc_function_handle_t handle[]; grpc_function_handle_init(handle[pixel],host_name,port_name,”lib_name”); } for (pixel 1 to 16) { //for the 16 pixels a asyncnous call occure grpc_call_async(handle[pixel], args….); } // Wait until every node finished grpc_wait_all(); for( pixel 1 to 16) { // all handles need to detruct grpc_function_handle_destruct(handle[pixel]); } grpc_finalize(); }
GA_SWAP(pixel_buffer, result){
Load(pixel_buffer);Run(swap);Send(result);
}
19
Results & DiscussionsResults & Discussions
• Finish only pixel distribution over cluster
• Experiment run on 3 clusters in the Grid – MAEKA 31 nodes Opteron Dual 1.4&1.8 GHz– AMATA 14 nodes Athlon XP 1 GHz– MAGI 4 Nodes Athlon XP 2500+
20
ResultResult
Pixelversion speedup
0.000
1.000
2.000
3.000
4.000
5.000
1 2 3 4 5nodes
speedup
pop10pix5
pop20pix5
21
ResultResult
Popversion speedup
0.000
2.000
4.000
6.000
8.000
10.000
12.000
14.000
16.000
18.000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
nodes
speedup
pop10pix1
pop10pix5
pop20pix1
pop20pix5
22
ResultResult
speedup 20 pop 5 pixels
0.000
2.000
4.000
6.000
8.000
10.000
12.000
14.000
16.000
18.000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
nodes
speedup
pixel-version
pop-version
23
Lesson LearnedLesson Learned
• Grid programming is still difficult – Learning curve– Debugging? – Grid reliability. May be useful to have a small
development grid separate from deployment grid
• Ninf-g application on grid is working fine. A good approach for development of grid applications.
24
Future WorkFuture Work
• Improve performance
• scheduler and node selecting (now is only round-robin)
• Test running across international Grid– collaboration with Prof. Kento Aida,Titech– PRAGMA testbed
• Deployment of ninf-g for Thai National Grid application development
Thank youThank you
Q&A