grid platform for geospatial applications & fine granule scheduler presented by bin zhou bin...
DESCRIPTION
Grid Computing Introduction ➲ Definition Grid computing is an emerging computing infrastructure that treats all resources as a collection of manageable entities with common interfaces to such functionality as lifetime management, discoverable properties and accessibility via open protocols – wikipedia ➲ Popular Grid Middleware Condor Globus Condor-G UnicoreTRANSCRIPT
Grid Platform for Geospatial Applications
& Fine Granule Scheduler
Presented by Bin ZhouBin Zhou, Jibo Xie, Chaowei Yang
Joint Center for Intelligent Spatial Computing
George Mason University
Agenda➲ Grid Computing Introduction➲ CISC & SURA Grid➲ Geospatial Applications Require Grid➲ CISC Fine Granule Scheduler➲ Architecture,Strategy➲ Progress Status
Grid Computing Introduction➲ Definition
Grid computing is an emerging computing infrastructure that treats all resources as a collection of manageable entities with common interfaces to such functionality as lifetime management, discoverable properties and accessibility via open protocols
– wikipedia➲ Popular Grid Middleware
Condor Globus Condor-G Unicore
CISC_Grid ManagerAdministrator
Ethernet
NASA
GMU
UserHigh Speed Connection
SURA Grid
Other Universities Virtual Organization
1G bits/s(can be updated to
10G bits/s)
CISCGrid Portal
Worker
CISC Computing
Pool
Worker Worker
KeyServer
GMUCertificate
Server
CertificateServer
ManagerFile Server
GMU grid environment
• SURAgrid
GMU
CISC GMU Grid can access the computing resources contributed by SURAgrid member universities
GMU grid environmentLambdaRail
GMU
CISC Grid can setup 1-10Gbps connection to any of the LamdaRail supported Universities, Agencies, and Centers, such
as GSFC & SDSC
CISC Computing Pool
Geospatial Requirements➲ Large Data Set
Map Data, Sensor Data, in Tera-bytes➲ Reliability,Interoperability
collaboration➲ Intensive Computation
More Complex Algorithms Adaptive Algorithms Intelligent Processing
Grid Computing Could Satisfy these requirements
➲ Reliable File Transfer➲ Resource Management and Allocation➲ Authorization & Control➲ Job Control➲ Web Service Oriented
Detecting Watersheds from multi-scale DEM
➲ Watershed boundaries are not known before processing massive data
➲ extract coarse watershed boundaries from multi-scale DEM ➲ Using the boundaries to decompose the massive data with
some redundancy
resample
Extraction
Xie 2006
Use 24 units to test the speed up
(each unit is 3.08M)
(Xie 2006)
CISC Test Applications
30 30
20 30
322s 293s
5.2 5.75
0.26 0.19
30
10
374s
4.5
0.45
Job Amount
CPUs
Executing TimeSpeed Up
Efficiency
30
1
1686s
1
1
Real Time Routing Test Result:
The efficiency decreases with the CPU numbers because the overhead increase, but the major problem is Condor can’t handle small jobs efficient.Demonstrates the need for fine granule scheduler
Specific Applications: Fine-Grained Near Real Time Jobs
➲ Fine-Grained Very Short Executing Time Huge Amount Job Similarity
➲ Near Real Time Sensitive to scheduling latency example: Real-Time Routing, Short-Time stock
prediction,Condor cannot be used for tasks that require less than 3.5 min to
complete ---Gregg Cooke, IT Technical Council ,"Evaluating Condor for
Enterprise Use: A UBS Case Study"
CISC Scheduler ➲ Purpose
improve near real time job response time improve mass Fine Granularity job throughput
➲ Scheduling Strategy Short Communicating Message Simple Match-Making Function Dynamic Index Multi-Dispatch
System Architecture
TCP/UDP SocketFile Transfer Process Other
LibServices
Abstract Interface /APIs
Message passing Memory
System Function
Dispatcher
Collector
Container
Resource Manager
Submitter
Algorithm module
Central ManagerWorker User Interface
ComponentsServer Daemon
Scheduler
Collector
Dispatcher
Dynamic IndexJob Queue
Machine Queue
CPU
Poll orEvent Driven
Multi-Dispatch
Priority Deadline
Sort
Client Daemon
Submittor
Resource Manager
Job Package
File Fetcher
Worker Status
Resource Info
ShieldLocal
Running Env
Match Queue
Dynamic Index
Dependency
Rank
Availability Memory Disk Others
Job&Workerinfo
Job Parser
Job File Server…...
Job Work FlowSubmit
InQueue
Schedule
Running
Dispatch
Staging out
Finished
Resource
Scheduling Algorithm
RemoveFrom
Queue
If Error
If Error
Prototype Overhead Test
➲ Test Case Insertion Sort 200,000 integers Dataset: 5.56M Execute File : 1.8M
➲ Test Platform OS: ubuntu 6.10 Network: 100Mbps CPU: Celeron M 1.6G Memory: 1G
Job Amount
File Transfer Time
Job Executing Time
Other Overhead
CommunicatingOverhead
Efficiency
1 1s 27s 0.4s 18ms 95.1%
5 4s 154s 1.2s 20ms 98.9%
Thanks
Questions?