towards developing next-generation distributed software a society-inspired methodology
DESCRIPTION
Towards developing next-generation distributed software A society-inspired methodology National University of Defense Technology, China Bo Ding [email protected]. Outline. Challenges ahead Key concepts of our methodology Autonomic Unit and Commonwealth - PowerPoint PPT PresentationTRANSCRIPT
Palette - WP0
European and Chinese Cooperation on Grid
Towards developing next-generation distributed softwareA society-inspired methodology
National University of Defense Technology, ChinaBo Ding
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Outline
Challenges aheadKey concepts of our methodologyAutonomic Unit and
CommonwealthFrom concepts to real-life programmingA reference
software architectureA prototype of software
infrastructureConclusion & future work
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Grid technologies: state-of-the-art
Large-scale resource sharing and
virtualizationApplicationsFramework and
InfrastructureArchitectureToolsImpact on practice of science and
industryECHOGRID: Towards a Shared EU & Chinese Vision for Grid
Research PerspectivesOther related research projects:China 973
program Research on aggregation and collaboration mechanisms in the
virtual computing environmentChina 863 program Research on
self-adaptive software platform for pervasive computing
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Distributed software is becoming more and more complexRunning on
millions of computing nodes, sensors and actuatorsBeing sprayed
everywhere and highly interconnectedTypical instances which is
turning into reality:Internet-based large-scale computing
systemPervasive computing application
A gradual revolution
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Characteristics & Related work
Characteristics of next-generation distributed
softwareUbiquitySeamless integration of the physical and computing
spacesDecentralizationHuge complex systemEach node is inherently
autonomousContinuous GrowthRequirements, functions and physical
boundary are dynamically changing in the whole lifetime of the
softwareHow to develop such kind of system has been a great
challengeA new focus in the research communitySoftware-intensive
systemsUltra-Large-Scale systems (CMU/SEI)Spray computer
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Challenges of development
Some principles of developmentEvolution, instead of waterfall
modelCooperation, instead of centralized controlActive adaptation
instead of being passively invokedMethodology have to be proposed
and software infrastructure have to be developedTo fulfill the gap
between our ideal and traditional software engineering
technologies
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Outline
Challenges aheadKey concepts of our methodologyAutonomic Unit and
CommonwealthFrom concepts to real-life programmingA reference
software architectureA prototype of software
infrastructureConclusion & future work
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Inspiration from social behaviors
Society is a typical huge open complex systemEach individual is
autonomicHe continuously and actively adapts to the environmentEach
individual belongs to one or more organizationsHe enjoy rights as
well as undertake duties, for instance, having to abide by the
organizations rulesThe progress of the whole society is driven by
the cooperation of active individuals, whose behavior is regulated
by certain organizations rules.A brief overview of our
methodologyResources are abstracted to Autonomic Units
individualsAutonomic Units can dynamically join/quit Commonwealth
organizationVarious task targets are achieved by the cooperation
between regulated Autonomic Units.
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Autonomic Unit and Commonwealth
Autonomic Unit (AU)Software abstraction of concrete resources, who
has the capability toJoin: Becoming a member of Commonwealth, being
able to abide by Commonwealth's rulesAdapt : actively adapt to the
physical and computing environmentCommonwealth (CW)Consisting of
Autonomic Units having the same interests or targetsEnacting
behavior rules to ensure interoperation and encourage equal
cooperation
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Abstraction
Autonomic Units
Software Space
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Model of AU & CW
Autonomic Unit
Commonwealth
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Behavior-driven Engine
ContextCollectors
BehaviorExecutors
Physical and computing space
Autonomic Unit
Base-level entities
Meta-level entities
Commonwealth Join/Quit Mechanism
Abstraction of sensors, which collects computing and physical
contexts
Concrete business logic, whose execution may change internal states
or the envrionment
Actively driving the behavior based on the change of concerned
contexts
Dynamically join and quit a Commonwealth, ensuring AU being
regulated by the rules of the Commonwealth
Initiator AU
Common AU
A special AU to initiate a Commonwealth
AUs dynamically join/quit the Commonwealth
Commonwealth
Resource Discovery
To help initiators to invite appropriate AUs under certain
circumstances
Second EchoGrid Workshop Beijing 29 & 30 October 2007
How to address the challenges?
? Evolution, instead of waterfall modelBehavior dynamically
regulation instead of static deploymentThe developers can specify
and modify the rules of Commonwealth on lineAU gets the newest
rules by joining CommonwealthDeveloper-driven system dynamic
evolution? Cooperation, instead of centralized controlEncourage
cooperation by Commonwealth rules? Active adaptation, instead of
being passively invokedContext-driven and reflection-based
adaptationIn brief, Join/Adapt capability partially addressed the
challenges to the development of next-generation distributed
software!
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Outline
Challenges aheadKey concepts of our methodologyAutonomic Unit and
Commonwealth From concepts to real-life programmingA reference
software architectureA prototype of software
infrastructureConclusion & future work
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Component-version Autonomic Unit
To integrate AU concept with maturing software engineering
technology!Component-based designDownload-based joiningRule-based
adapting
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
BehaviorComponent
Behavior-drivenengine
Policies
AUShell (Container)
BehaviorComponent
BehaviorComponent
A set of adaptation rules, in the shape of when to do what
To translate and execute the policies
ContextComponent
ContextComponent
Encapsulation of context providers
Encapsulation of business logic
Physical and computing space
AU Core
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Dig deep into Join
Question: How to ensure AUs behavior regulated by the
Commonwealth?Answer: Download-based JoiningAn Autonomic Unit is
divided into two parts:AU Shell: Container, including
behavior-driven engineAU Core: Commonwealth-related objects,
composed of policies and componentsAU Shell is pre-deployed and
loaded in the start-up process of each resourceAU Core is
dynamically downloaded when an AU joins a Commonwealth
Concrete joining model: Actively joining model, Inviting Model, and Mixed Model
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Container(Behavior-driven engine)
PolicesBehavior componentsContext components
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Question: How to actively adapt to the environment? Answer: Rule-based AdaptationFrom the perspective of reflectioncontext component meta data providerpolice meta-level protocolbehavior-driven engine meta-level computing entitybehavior component base-level computing entity.
ComponentManagement
Dig deep into Adapt
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
RequestDispatch
Comp
Network Message
Management Request
Behavior-driven engine
Policies
Context
Comp
Invocation/management
Traditional Container
Our AUModel
Second EchoGrid Workshop Beijing 29 & 30 October 2007
A brief overview of our software architecture
Concept view
A complete light-weight component model is proposed.Dynamically
loading, inversion of control, language independenceRun-time
viewInitiate: Commonwealth is initiated by an initiator AU, and
other AUs dynamically join/quit AU based on their interests and the
demandCooperation: AUs are cooperated driven by the context,
including the change of physical environment, computing resources
and network state.Development viewTo be continue in the next
slide
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
component
Autonomic Unit
Common Wealth
Second EchoGrid Workshop Beijing 29 & 30 October 2007
How to develop?
Step1: To define component interfaces by our UCDL languageStep2: To
implement components
Step3: To assemble CommonwealthAssemble components in a graphical tool based on GMEDivide assembled components into AU CoresIf necessary, specify policies in the AU Cores
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Design of software infrastructure
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Protocol Framework
Abstraction of Communication
Routing/Fowarding
ProtocolSwitching
NetworkStateMonitor
XML Parser
Meta-level interaction
AU Context Management
Binding
AU LifecycleManagement
Policy Execution
ComponentManagement
Initiator
Human-machineInteractionAdaptation
Context-awareData Storage
Event Service
Resource Discovery
AU Shell
Self-adaptiveCommunication
CommonServices
UCDLCompiler
ComponentPackager
AssmblingTool
ManagementTool
Run-time Environment
Development/ManagementTools
Protocol Plugins
ComponentLibrary
Infrastructure prototype and demo applications
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Outline
Challenges aheadKey concepts of our methodologyAutonomic Unit and
CommonwealthFrom concepts to real-life programmingA reference
software architectureA prototype of software
infrastructureConclusion & future work
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Conclusion & Future work
Our first step towards next generation distributed softwareProposed
an abstract model to partially cope with challenges aheadIntegrated
our ideal with maturing software engineering technologyFuture
workDynamic system theories and their application in our
methodologyEmergent Behavior, Digital evolution, Control
TheoriesSocio-Economic theories to encourage equally
cooperatingGame rules, Mechanism design, Credit countOther possible
self-adaption and behavior regulating mechanismsFurther research on
lightweight component model, policy and QoS integration
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
Thank you for your attention!
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Second EchoGrid Workshop Beijing 29 & 30 October 2007
*
Good morning, everybody!Its a pleasure to have the opportunity to
introduce our recent work here. Today I would like to present our
initial attempts towards developing next-generation distributed
software, especially our society-inspired methodology and software
infrastructure.
Ill give this talk in five parts: At first, well start with a brief
review of the challenges to the development of next-generation
distributed software. And then Ill present our methodology,
focusing on some key concepts such as Autonomic Unit and
Commonwealth. The third part of my presentation concerns how to
integrate our idea with existing maturing software technologies,
including a reference software architecture and a software
infrastructure prototype we developed to support this architecture.
If time permits, Ill run a simple demo on my notebook. And the last
part is a brief conclusion and introduction of our future
work.
*
*
As we know, the main purposes of the Grid computing technologies
are concerning about large-scale resource sharing and
virtualization. During the past decade, many research works have
been carried out within this community regarding to Grid
applications, frameworks and infrastructures, architecture and
tools. They have had tremendous impacts on the practices of science
and industry as well. However, because of the diversity of the
world we are facing today, the Grid people have also had a lot of
distinct characteristics in their research works. For example,
different viewpoint, different focuses, different modeling,
different methodology of implementation, to name a few. Is this
situation what we want? I think the answer is yes and no. Anyway,
for the time being, trying to move towards a shared vision for Grid
research is the main topic, which is also the reason of this
workshop. In the following slides, I would like to introduce our
vision on methodology and infrastructure within the context of
large-scale distributed software, or Grid, for short.Please note
that this work is also partially supported by some other related
projects. I listed the most important two of them here.The first
one is funded by The National Basic Research Program of China (also
called 973 Program), its main target is about aggregation and
collaboration mechanisms in the virtual computing environment,
which is also an very important topic in Grid.The second one is
funded by The Chinese High-tech program, its feature is on the
self-adaptation and pervasive aspects.
Okay, lets start off with challenges ahead. Over the past few
years, we have witnessed a rapid growth in size and complexity of
distributed software, which will likely continue well into the
foreseeable future. A software system may run on thousands of
computing nodes, sensors and actuators. Software entities will be
sprayed every corner of our lives and highly interconnected by
various networks. This is not science fiction. They are coming into
our reallife. Pervasive computing applications and Internet-based
large-scale computing systems are typical instances, such as
pervasive health, intelligent traffic and military information
system.
*
The growth in size and complexity is a gradual revolution.
Quantitative changes leads to a fundamental change. In another
word, those systems presented some characteristics that their
predecessors dont have, including: Ubiquitous, that is, seamless
integration of physical and computing space, to provide services
anytime, anywhere and invisibly;Decentralization, on the one hand,
the system is too complex to be managed and controlled centrally;
on the other hand, the nodes of such system may belong to different
domain, and thus they are inherently autonomous.Continuous growth,
that is, the requirements, functions as well as physical boundary
of the system cant be determined in advance. They are dynamically
changing in the whole lifetime of the software.Because of those new
characteristics, how to develop such kind of distributed software
has been a great challenge and aroused the interests of the
research community. For example, the research on software-intensive
system, being referred to as Systems in which software interacts
with other software, systems, devices, sensors and with people; In
July 2006, founded by U.S. Department of Defense, Carnegie Mellon
University published the report Ultra-large-scale systems,
presented a research agenda for Ultra-large-scale systems. Somebody
believes that this report is as important as the software
engineering report published in 1968.
*
Lets consider challenges ahead in more detail. Because
next-generation distributed software is ubiquitous, decentralized
and continuously growing, it is obvious that several fundamental
changes in the development should be taken place: Firstly, in order
to fit in with the continuous growth of the software, traditional
water-fall model should be abandoned and evolution in the whole
lifetime should be encouraged. Secondly, because of the
decentralization of such systems, traditional centralized control
and management is nearly impossible, we should encourage equally
cooperation between software entities instead. Thirdly, for each
software entity distributed on the computing node, we should
encourage them to actively adapt to the physical and computing
environment instead of merely being passively invoked.However, the
above-mentioned changes are just our ideal. It is obvious that
there is still a huge gap between it and existing software
technologies. In the following slides, Ill introduce our initial
attempts to fulfill this gap.
*
The second part of my presentation is about our methodology.
However, Ill just lay emphasis on some key concepts, partly because
of the time limit, partly because the research is still under
way.
*
Next-generation distributed software is supposed to be huge open
complex system. Lets take a look at a real huge open complex system
at first at first, the human society. In a society, each individual
is autonomic, continuously and actively adapting to the change of
the environment. Each individual belongs to one or more
organizations. The organization have clear targets, and as a member
of this organization, individuals enjoy rights as well as undertake
duties, for instance, having to abide by the organizations rule.
For example, if a man want to gain higher education, he must become
a student of a college and learn the rules of the college,
including communicating with teachers and classmates in an
appropriate manner, gaining sufficient credit, going to the lab on
time, and so on. The progress of the whole society is driven by the
cooperation of active individuals whose behavior is regulated by
certain organizations rules.To some extent, our methodology is a
simulation of human society. In our model, devices, who possesses
various computing resources, are abstracted into Autonomic Units,
an analogy of individuals tin he society. Autonomic Units can
dynamic join or quit Commonwealth, an aggregation of Autonomic
Units who have same interests and targets. Commonwealth is an
analogy of the organization. Various targets are achieved by the
cooperation between Automatic Units, whose behavior are regulated
by certain Commonwealth. This is just a brief overview and Ill
elaborate on those concepts in the next two slides.
*
Autonomic Unit, as we have said, is the abstraction of device. This
abstraction provide at least two important capabilities: join and
adapt. The former is referred to as its capability of joining a
Commonwealth and abide by its behavior rules; the later means that
it can actively adapt to the change of physical and computing
space. Commonwealth is an aggregation of Autonomic Units who
dynamically join and quit based on their own interest and the
demand. It enact some behavior rules to ensure interoperation and
encourage equal cooperation by the means such as credit count and
peer to peer resource sharing. The following animation illustrates
those concepts in more detail.Various resources is abstracted into
a software entity named Autonomic Unit with join/adapt capability.
When Autonomic Unit joins a Commonwealth, it becomes active. This
is the software space. There are already several inactive Autonomic
Unit, here inactive means that they dont join any Commonwealth at
this time. When a task is given, an Autonomic Unit may initiate a
corresponding Commonwealth, and other Autonomic Units may join this
Commonwealth dynamically. As a result, the boundary of Commonwealth
is not fixed. Another commonwealth may be organized in the same
style.
*
This is the internal structure of Autonomic Unit. As shown in this
diagram, it consists of four parts: Context Collector, Behavior
executor, Behavior driven Engine and Commonwealth join
mechanism.This is the structure of Commonwealth. A commonwealth
consists of two kinds of Autonomic Units: a special Automatic Unit
can act as an initiator, and other Autonomic Units dynamically
join/quit. Besides, a resource discovery service is provided to
facilitate the initiator to found appropriate AUs under certain
circumstance.
*
Okay, this is an outline of our methodology. lets go back to the
question: how does our methodology address those challenges to the
development of next-generation distributed software? We have
present three challenges earlier: evolution, cooperation and active
adaptation. The evolution in our methodology is implemented by the
Join capability, that is, behavior dynamically regulation instead
of static deployment. The developers can specify and modify the
rules of Commonwealth on line, and Autonomic Units can get the
newest rules by joining Commonwealth. In summary, on the macro
level, this is a developer-driven system dynamic evolution. With
regard to cooperation, it is obvious that our methodology is based
on a naturally decentralized architecture and Ive mentioned that we
can encourage cooperation by Commonwealth rules. In order to
implement active adaptation, we introduced Context-driven and
reflection-based adaptation, as shown in the concept model diagram
of Autonomic Unit.
*
Up to now, we mainly focused on abstract concepts and models. If we
cant integrate them with existing software engineering
technologies, it is just an empty talk. Therefore, well propose a
reference software architecture based on maturing component
technology and introduce a prototype of software infrastructure we
developed to support such an architecture.
*
This is the component version Autonomic Unit, an implementation of
the concept model we proposed earlier. In this implementation,
context collectors and behavior executors are encapsulated as
components, which we called context component and behavior
component. And a new entity, policy, is introduced to ensure
rule-based adaptation. Policy is a set of adaptation rules in the
shape of when to do what and can be dynamically modified by human
or other software. And the Behavior-driven engine is embodied as a
fixture to translate and execute policies.You may have noticed the
word downloaded-based joining and Rule-based adapting. Yes, this is
concrete the implementation of join/adapt capability. Ill explain
them at length in detail.
*
This slide explains the join mechanism in detail. As what Ive said,
AU is divided into two parts, AU shell and AU Core. The Shell is
general-purpose objects in an Autonomic Unit, mainly the Container
including behavior-driven engine. It is fully implemented in the
software infrastructure and pre-deployed on each resource. The Core
is composed of Commonwealth-related objects, including policies,
context components and behavior components. It is dynamically
download when an AU joins a Commonwealth. Dynamic download is the
analogy of learning rules in human society, and by this mean, we
ensure AUs behavior regulated by the Commonwealth. Weve introduced
several concrete joining models in the reference software
architecture, including actively joining model, inviting model and
mixed model.. However, in the interest of time, I wont go enter
into details.
*
Okay, Lets see how to implement the adapt capability? How to
actively adapt to the environment? The answer is rule-based
adaptation. From the perspective of reflection, .In summary, we
enhanced the meta-level function of containers in traditional
component model. This is a comparison of traditional distributed
component model and our Autonomic Unit Model. As shown, traditional
container is just a passive mediator of network messages and
management requests. On the contrary, the container in our
component-based Autonomic Unit model is able to actively manipulate
the base-level components according to current policies. Components
and Container are integrated into an organic whole.
*
Okay, its time for a brief view of the whole reference software
architecture. We originally prepared to introduce it from three
aspects: concept view, which explains key concepts of our
architecture; run-time view, which explains the running mechanisms
of software; and development view, which explains how to develop
step by step. However, in fact, in previous slides we have
explained concept view and run-time view. The only thing I have to
add is that the component model is a light-weight model we
proposed, which characterized by dynamically loading, inversion of
control and program language independence. However , I think this
is not the focal point of todays presentation. Since todays topic
is programming paradigm, Ill discuss our development view in more
detail.
*
The development of a Commonwealth can be divided into three steps.
The first two step is similar to traditional component development,
as shown in this slide. The first part of the third step is still
similar to traditional component composition, in an integrated
graphical tool we developed based on GME. The main difference is
that after assembling a Commonwealth, they have to divide the
Commonwealth into AUCores and indicate some necessary properties of
those Cores. And then the programmer can decide whether or not to
add policies to an AUCore. Please note that policy is optional. If
you chose to not to specify any policy, the development process is
very similar to traditional component-based application
development. We intentionally kept this compatibility with
traditional development method, hoping our method can be widely
accepted by the programmers.
*
We have outlined the reference software architecture, and now well
take a glance of our software infrastructure to support the running
and development of this architecture. This is the design blueprint
of our software infrastructure, which consists of Self-adaptive
communication, Autonomic Unit Shell, Common services as well as
development and management tools. Up to now, we have implemented a
minimal set of this blueprint, which can verify and demonstrate our
idea.
*
This is some Snapshots of the user interface of our infrastructure.
And we developed several applications based on this infrastructure,
to verify and demonstrate our idea. This is one of them, a fire
alarm demo. The left diagram is its assemble view, and the right
part is its run-time snapshot.
*
Okay, The conclusion and our future work. This is Our first step
towards next generation distributed software. In brief, we
*