chapter 4 proposed grid network monitoring...
TRANSCRIPT
39
CHAPTER 4
PROPOSED GRID NETWORK MONITORING
ARCHITECTURE AND SYSTEM DESIGN
This chapter discusses about the proposed Grid network monitoring
architecture and details of the layered architecture. This chapter describes the
design of the proposed system, the design of an automated deployment of the
proposed system as a service in Grid, the network metrics used for
performance evaluation and Network Cost Function. This chapter also
describes about the computation of Resource Cost Value using resource
metric, and Network Cost Value using network metrics. The Network Aware
Resource Selection strategy is also explored in this chapter.
4.1 PROPOSED ARCHITECTURE
The complex system like grid, monitoring is essential for
understanding its operation, debugging, and failure detection and also for
performance optimization. Due to the heterogeneity and constantly varying
nature of grid, estimation of network performance is indispensable. If the
status of the network path can be predicted, it is possible to use that
information in grid applications. For example, the network status can be used
to adapt the traffic load in order to avoid congestion on a network path.
Scheduling of large data flows for data intensive applications is highly
dependent on network path characteristics. For computationally intensive
applications, resource broker or scheduler needs to have comprehensive and
40
accurate knowledge of network properties to fulfill service level agreements,
ensure QoS, and to make fit choices for advance reservation.
The four layered Grid network monitoring architecture is proposed
and modeled with the grid scheduler in the collective layer. The proposed
architecture is based on OGSA compliant layered architecture which is shown
in the Figure 4.1. In the proposed approach, the CARE Resource Broker
(CRB) is used for job submission (Thamarai Selvi et al 2009).
Figure 4.1 Grid Network Monitoring Architecture
Fabric Layer
The Grid fabric layer defines protocols for the publication,
discovery, negotiation, monitoring, accounting and payment of the operations
41
on individual resources. The resources may be computational resources,
storage systems, catalogues, network resources and sensors or may be a
logical entity, such as a distributed file system, computer cluster, or
distributed computer pool. The Grid Resource Access and Management
(GRAM) protocol is used for allocation of computational resources and for
monitoring and control of computation on those resources, and Grid File
Transfer.
Resource and connectivity layer
This layer consists of low-level middleware that provides secure
and unified access to remote resources. Depending on the type of resources,
different middleware can be chosen such as Globus, Unicore, Alchemi, and
Storage Resource Broker. Using services of such low-level middleware layer,
one can create high-level middleware services that support rapid creation and
deployment of applications on global Grids.
Collective Layer
The proposed architecture is modeled in this layer with a grid
scheduler. The Request handler which resides in CRB receives job requests
from the users. The controller is in CRB which controls the scheduling,
selection of the suitable resource for job submission from the matched
resource list, the monitoring the execution of jobs in Grid, and also maintains
the status of the submitted jobs. CRB selects the suitable resource using
resource metrics. The network monitoring is fit in this layer to retrieve the
network metrics which have influence on the resource selection.
More sensors are deployed in grid resources to provide more
network metrics so that the measurement of the network performance
becomes more reliable. The sensors are the network monitoring tools and
42
utilities which are started through migration of mobile agents from resource
broker to all resource sites when there is a need of unplanned monitoring in
the Grid environment. The planned network monitoring gathers the network
metrics from information repository, because the sensors are the network
monitoring tools which are initiated on all the Grid resources to retrieve the
network metrics periodically and update the information repository. The
network performance measurement and prediction utilizes the information
repository to measure and predict the performance of the Grid.
Network Monitor monitors the network and collects the network
metrics such as bandwidth, RTT, packet loss, jitter and stores it into
information repository using agent based information aggregator. The
Resource Monitor monitors the grid resources and collects the resource
metrics and stores the collected information to information repository with the
aid of agent based information aggregator. The agent based information
aggregator aggregates the resource and network information from the Grid
resources and periodically updates the information repository. It maintains the
information about every physical resources and its performance of the
network through end to end network monitoring across the grid infrastructure.
The data manager uses the network cost function to measure the network
performance which is described in the section 4.4.
CRB catalogs the matched resources depends on the job
requirements submitted by the user. Resource selector queries the information
repository to select the suitable resource with network aware resource
selection strategy and sending that information to the scheduler. The job
monitor maintains its current status of the job execution and reports the
progress to the user through Resource Broker. The network predictor predicts
the future network performance using History Based (HB) approach and
stores the predicted values into the information repository. It is also
43
responsible for sending the predicted value to the scheduler to take complex
decisions.
Application layer
The application layer facilitates the use of resources in a grid
environment through resource access protocols. The portal present at this
layer allows the grid user to submit resource requirements to find out suitable
resources for the execution of the submitted applications. It also includes
software and tools to support application workflow and composition.
4.2 DESIGN OF THE GRID NETWORK MONITORING SYSTEM
The proposed Grid Network Monitoring System design is based on
the architecture described above. The Figure 4.2 provides the view of the Grid
Network and Resource monitoring at resource level.
Figure 4.2 Grid Network and Resource Monitoring at Resource Level
44
Grid Network Monitor
The Grid network monitor initiates measurements or predictions on
demand. The sensors are deployed to provide more network metrics which
provides reliable network performance. The sensors are network monitoring
tools and utilities like UDPmon, TCPmon, IPerf and Ping. These sensors
provide network metrics like bandwidth, RTT, packet loss, and jitter which in
turn facilitates the network status monitoring.
Mobile Agent Generator
In Grid, whenever the user submits the job through the resource
broker, the mobile agent is created from mobile agent generator and it is
cloned and migrated to all grid resources and starts the sensors. The sensors
are the network monitoring tools which are used to retrieve the network
metrics between the end-to-end node in all grid resources.
Resource Monitor
The mobile agent migrates from the resource broker to all resources
and collects the resource metrics. The Resource Monitor monitors the resource
metric and sends the monitored information to data collector. The data
collector aggregates the collected information and periodically updates the
local archive maintained in every CE.
Data Accumulator
The Grid network monitor initiates data accumulator to collect the
metrics. In the Resource Broker and Grid Resources (i.e. head node), the
server for all the sensors running, from that the network metrics for the link
between the head to each compute node is sent to data accumulator. The
accumulator extracts the necessary data from multiple compute nodes and
45
stores it in global archive, also called as information repository present in the
head node. The resource metrics are collected from each node along with IP
address of the corresponding node and a time stamp to maintain validity for
the data. The agent performs this task for all computed nodes in the grid
cluster and updates the global archive.
Data Processor
The data processor process the network data collected through
deployed numerous sensors to measure the network performance using
network cost function. The network cost value and resource cost value is
calculated for all Grid Resources, through which the compound cost value is
calculated to help the scheduler for selecting the suitable resource for job
submission.
Global Archive
It contains aggregated information about network metrics and
resource metrics of all Grid Resource. The Global archive, also called as
Information Repository resides in the head node. This also stores the results of
the cost function computations which are used by the data processor to
measure the network performance and also for prediction.
Predictor
The predictor is linear and Historic-Based. This model uses
standard time series forecasting techniques to predict the performance based
on a history of measurements from previous behaviors on the same path. In
the proposed system, Holt-Winters (HW) model is used for predicting
network performance in the near future.
46
Visualizer
This part deals with displaying the monitored network
characteristics and the predicted network performance. The deliverables from
this component may be a graph or a chart providing a clear vision about the
network status of all grid resources.
Sensors
This component deals with the actual data collection in the Grid
Resources. The numerous sensors are deployed using network monitoring
tools and utilities like UDPMon, TCPMon, IPerf and Ping. These sensors
provide metrics like bandwidth, latency, packet loss rate, jitter, round trip
time (RTT) and one-way delay which in turn facilitates for network status
monitoring. The communication among the model components is depicted
in Figure 4.3.
Figure 4.3 Work Flow of the Proposed Monitoring System
47
4.3 NETWORK METRICS FOR PERFORMANCE
EVALUATION
Monitoring the Grid network performance requires the analysis of
various parameters like bandwidth, RTT, packet loss, jitter, etc which varies
frequently depending upon the real time network conditions across the links.
These parameters individually won’t determine the network performance
accurately. Thus combinations of the parameters are required.
4.3.1 Bandwidth
The maximum amount of data per time unit that a hop or path can
provide given the current utilization. The available bandwidth of a link relates
to the unused, or “spare”, capacity of the link during a certain time period. At
any specific instant in time, a link is either transmitting a packet at the full
link capacity or it is idle so the instantaneous utilization of a link can only be
either 0 or 1. Thus any meaningful definition of available bandwidth requires
time averaging of the instantaneous utilization over the time interval of
interest. The average utilization, u(T-t , T) for a time period (T-t, T) is given
by u(T-t , T) = (u(x) ) over the limit T-t and T. where u(x) is the
instantaneous available bandwidth of the link at time x. IPerf deals with TCP
bandwidth and UDP bandwidth.
Bandwidth = S / ( T – Latency) (4.1)
where, S is the message size, T is the message transfer time, and Latency is
measured from RTT.
4.3.2 RTT
Round Trip Time (RTT) is the time at which the last packet byte
departs from the source t(D), and the time at which the last packet byte arrives
48
at the packet destination t(A). The Ping utility finds an application to measure
this metric.
RTT = t (A) - t (D) (4.2)
4.3.3 Jitter
Jitter is the "instantaneous packet delay variation" (IPDV) and it
denotes the difference experienced by subsequent packets, I and I+1, on a
one-way transit from source to destination. Iperf is used to measure it.
4.3.4 Packet Loss
Packet Loss indicates the percentage of loss of data packets when
the packets are transmitted between the end hosts. Packet loss may take place
due to hardware fault, congestion in the channel, corruption in the data packet
sent. The data packets that are discarded by the routers when the load
becomes heavy also accounts for the packet loss percentage. Iperf and
UDPmon Tools are used to measure packet loss rate.
Packet loss percent = {1 - (Received Acks / Sent Packets)} * 100 (4.3)
4.4 NETWORK COST FUNCTION
The Measurement of the network characteristics like latency,
throughputs, packet loss rate, jitter, etc, is a repeated operation in any network
management system and also in Grid Environment. A single network
characteristic does not provide the significant information about the network
performance through network resources. So an aggregation of multiple
network metrics known as Network Cost Function (NCF) is needed to
measure the performance of the network. The metrics considered are
bandwidth, RTT, packet loss, and jitter between any nodes in the grid cluster.
49
The location of nodes could be inferred from the list of nodes available in
Grid.
The bandwidth measure is taken with respect to the average and the
maximum value. BWmax value is considered because of considering the
maximum possible bandwidth in the network channel. The variation of the
RTT values is very wide; hence there is a need of normalization. The half
normal form is used here, because the RTT values contain only the positive
values and this distribution is very specific version of normal distribution. The
packet loss, p and jitter are raised to the powers of the arbitrary values which
are decided based on the current Grid set up. The values of the arbitrary varies
[0, 1], hence the resulting values also reflect the values in [0, 1]. Packet loss
rate is significant in network cost functions because it provides an estimate of
both short and long-term congestion on a given data path due to packet drop
which depends on the performance of the transfer protocols. IPDV is an
important quality of service factor in assessment of network performance. If
there is no packet loss and jitter, the NCF influenced with bandwidth, and
RTT. The NCF (Network Cost Function) varies in the range [0, 1], where 0
indicates that a given node is not reachable and 1 denotes the maximum
degree of usage of the link if the network is congestion free.
Let BWmax denotes the maximum available bandwidth between
the corresponding pair of nodes.
Let BWavg denotes the mean available bandwidth between the
corresponding pair of nodes, i.e., BW = BW , where n is the
number of values taken for calculating the mean bandwidth for a period of
time.
50
Let RTTmax denotes the maximum RTT between the corresponding
pair of nodes, RTTavg denotes the mean RRT values measured by individual
probes between the corresponding pair of nodes, i.e., RTT = RTT ,
where k denotes the number of values taken for calculating the mean RTT for
the same period of time. And denotes how the RTT have influence on NCF.
= RTTavg RTTmax (4.4)
Let p denotes the packet loss, varies in interval [0, 1] and tuned to
balance the dependency of packet loss.
Let jitter denotes the measure of the variability over time of the
packet delay across a network, called as IPDV. varies in interval [0, 1] and
tuned to balance the dependency of jitter. The setting of , and are depend
on the Grid cluster which influences the maximum value.
The NCF of end-to-end node is measured by analysing the network
parameter values such as bandwidth, RTT, packet loss and jitter between that
pair of nodes. The NCF of the individual links are calculated using the
following expression.
NCF = × e × × (4.5)
4.5 RESOURCE COST VALUE
The computation of the Resource Cost Value (RCV) of the Grid
Site (GS) is based on the available Free Memory of the grid resources. The
average Free Memory, FreeMemavg is calculated for each Grid Resource in a
Grid environment, i.e., FreeMem = FreeMem , where, n is the
51
number of resources in a GS. The maximum available memory for a
Grid Site, i.e., FreeMemmax = max n (FreeMemk) is also identified to
evaluate the RCV, where n is the number of resources or Computing
Elements (CE) available in a Grid Site. The RCV is computed by the
following expression.
RCVGS=avg
max
FreeMem
FreeMem (4.6)
According to the Equation (4.6), the RCV of Grid resource, RCVGS
is varies in the range [0,1].
4.6 NETWORK AWARE RESOURCE SELECTION STRATEGY
The integration of network information with resource information
has very much influence in the decision making process of a Grid Resource
Broker. One of the major functions of the resource broker is to select the
suitable resource from the list of Grid resources which are geographically
distributed. The components of the network aware resource selection are
shown in Figure 4.4.
Figure 4.4 Network Aware Resource Selection Component
52
The primary selection is based on the requirements to be needed for
the execution of job. In the proposed approach, the CRB is used as a primary
selection to identify the matched grid resources for the specific job. The
primary selection rules are defined by the job when it is submitted from CRB,
which are the requirements needed to execute the job, such as software
needed to execute the job, CPU Speed, and Memory, etc. The primary
selection rules are used by the CRB broker to list the suitable resources to
execute the job which are available in Grid environment. The execution of the
job needs of transferring multiple input files and output data which lead to
produce the traffic across the Grid. The amount of traffic can be reduced
while selecting the node with better network connectivity. So there is a need
of measuring the performance of the network using network metrics such as
network cost value. The resource cost value is also computed by considering
the available free memory. The Resource Discovery uses the secondary
selection rule which is defined by the combination of network cost value and
resource cost value called as compound cost value to select a suitable node
from the CRB matched list of resources.
4.7 JOB MONITORING
The Job Monitor is responsible for maintaining the status of the
execution of the job to track its progress which is shown in the Figure 4.5.
Grid Resource Allocation Management (GRAM) is implemented as a Web
Services Resource Framework (WSRF) service in GT4 (Feller et al 2007).
GRAM provides an API that allows for submitting and canceling a
job request as well as checking the status of a submitted job. The job file is
written using JSDL. After the job file is given as the input, it is passed to
53
JobMonitor. The JobMonitor uses the security provided by the Grid Security
Infrastructure (GSI) of the Globus toolkit. WS-GRAM supports signature and
XML encryption. It uses digital certificates to send secure XML SOAP
messages between the Resource Broker and the Grid Resource.
Figure 4.5 Function of Job Monitor
Grid Resources have the local resource management system
(LRMS) which controls jobs running on CEs. It allocates CEs to jobs, starts
and stops jobs on user request and possibly restarts jobs if an error occurs.
The LRMS identi es the job it manages using local job identi er (LJID). The
jobmanager is the Globus GRAM which allows Grid users to start jobs on a
Grid resource.
Store
Credent ials
using RFT
Authent icate
Schedule
Job M onitor
Store
Report StatusReport StatusJob State
M onitor
Job Event
Daemon
Executes jobUser Job
Fork
Scheduler
Delegat ion
GRAM
Job Manager
User
Reports job status
Submits job file
54
The user submits a job manifest, a document which contains the job
description and the specification of the requested local resources to the
jobmanager. After successful authentication and authorization, the
jobmanager translates the job manifest into a form understood by the local
resource management system and starts the job under a local user account i.e,
user ID. A different user account is assigned to identify processes belonging
to a job is to start. Then the GRAM ensures that each job is started under a
user account that is distinct from accounts used by other presently running
Grid jobs.
First user delegation is performed, and then the job execution starts
at the CEs. Then the job status and progress is updated periodically at the
Resource Broker where the user submits the job. Once job execution
completes, the results are reported to the user along with any possible errors
like the job contact string does not match any which the job manager is
handling which are reported by GRAM. The JobMonitor also identifies the
current directory of the job, files in that directory, the permission set up of
those files and its resource consumption.
The Figure 4.6 depicts the process of job submission and
monitoring. The jobmonitor represents the Grid service, which allows Grid
users to start jobs on Grid resource.
Figure 4.6 Sequence diagram of job submission and monitoring process 55
56
4.8 NETWORK PERFORMANCE PREDICTION
There are three simple linear predictors namely Moving Average,
Exponential Weighted Moving Average (EWMA), and non-seasonal Holt-
Winters. These predictions have some order for prediction say ‘n’ based on
the number of previous values taken into consideration for the prediction. A
first order prediction is simpler but may not be accurate; higher the order then
the predicted value is more accurate. There are also more complex linear
predictors but selecting their order and linear coefficients requires a large
number of past measurements. So the simple predictor Holt-Winters (HW)
method is considered for predicting network performance rather than complex
ones.
4.8.1 History-Based Prediction
The History-Based (HB) prediction method is similar to traditional
time series forecasting, where past samples of an unknown random process
are used to predict the value of the process in the future.
4.8.1.1 Moving Average
Given a time series Y, the one-step n-order Moving Average (MA)
(n-MA) predictor is:
i+1 = (n)-1
{ Y(i-n+1)+Y(i-n+2)+..+Y(i-1)+Y(i) } (4.7)
where, i is the predicted value and Yi is the actual (observed) value at time i.
If n is too small, the predictor cannot smooth out the noise in the underlying
measurements. On the other hand, if n is too large the predictor cannot aptly
adapt to non-stationarities.
57
4.8.1.2 Exponentially Weighted Moving Average
The one-step Exponentially Weighted Moving Average (EWMA)
predictor is
X = X + ( )X (4.8)
where, is the weight of the last measurement (0< <1). Similar to the MA
predictor, a higher cannot smooth out the measurement noise, while a lower
is slow in adapting to changes in the time series.
4.8.1.3 Level Shifters and Outliers
While experimenting with various predictors, it was found that the
largest prediction errors are often caused by level shifts and outliers in the
observed time series. Furthermore, if there is a need of manage to avoid these
two characteristics in the time series forecasting, the exact choice of the
predictor, or of its parameters, does not make a significant difference.
A level shift is a type of non-stationarity, and it causes a significant
and typically sudden change in the mean of the observed time series. An
outlier is a measurement that is significantly different, beyond the typical
level of statistical variations, relative to nearby measurements. One way to
deal with level shifts, after they are detected, is to restart the predictor,
ignoring all previous history. Outliers, on the other hand, can be just ignored.
4.9 ARCHITECTURE OF THE AUTOMATED DEPLOYMENT
OF NETWORK AWARE RESOURCE MONITORING
SERVICE
The proposed architecture for an agent based Automated
Deployment of Network aware Resource Monitoring service is shown in the
Figure 4.7. The proposed mobile agent based automated deployment avoids
58
the maintenance costs and human errors occurring during deployment. Since
mobile agents are capable of operating even without active connections
between nodes, they are not affected by network failures. Further mobile
agents reduces network load. Mobile agent technology is very much flexible
to support the rum-time mobility through push and pull interaction models.
And the characteristics persistence, cloning and migration of the mobile
agents improve the reliability through replication.
Figure 4.7 Automated Deployment of Network aware Resource
Monitoring service
A mobile agent is composed of code and data which migrates to other
nodes and executes there in the node to which it migrates. The mobile agents
exploit the basic communication protocols defined within IBM Aglets
Workbench (Aglets 2004) for agent migration and to dispatch messages from
one node to another node. Deployment Agent is a mobile agent which contains
code for deployment of the services. It resides in the Resource Broker. On
request from a newly arrived Grid resource, it migrates to the Grid resource and
executes. The node where the Resource Broker is running is act as a Registration
Node which maintains a database of the Grid resources that arrive and also it
sends the IP address of the Resource Broker to the newly arrived resource.
The Resource Broker contains the services to be deployed and the
deployment agent. The registration node maintains a repository of IP
59
addresses of the resources in which the monitoring service is deployed and
also the IP address of the Resource Broker. Mobile agents are used to get the
IP address of the Resource Broker from the registration node and for
deploying the services. For an automated deployment of service, the mobile
agent migrates from the new resource to the registration node, collects the IP
address of the Resource Broker and migrates back. Using the IP address of
the Resource Broker, the newly arrived resource requests the Resource Broker
for deployment of the services. Then the deployment agent migrates from the
Resource Broker containing the services to the newly arrived resource and
executes thereby deploying the services.
4.10 SUMMARY
The monitoring of Grid resources is a momentous task because of the
diversity of the computing resources and applications in Grid environments. The
existing resource brokers are not considering the factors such as location of data,
bandwidth availability and data transfer time while scheduling data-intensive
applications on Grid Resources. This chapter presented a four layered
architecture for Grid network monitoring system which is modeled with Grid
scheduler. The Grid network performance is measured using Network Cost
Function by analyzing the network metrics such as bandwidth, RTT, packet loss
and jitter between the pair of nodes in Grid. The resource cost value is computed
using resource metrics and the network cost value is computed using network
metrics. The proposed system is integrated with CARE Resource Broker (CRB)
which is used for job submission. The Network Aware Resource Selection
Strategy is proposed for resource selection by computing the compound cost
value using network cost value and resource cost value for the selection of a
suitable node from the CRB matched list of resources. Once the job execution is
completed, the results or errors are reported to the user which is provided by
GRAM (Grid Resource Allocation and Management). The agent based
automated deployment of Network aware Resource Monitoring service also
explored in this chapter.