UNESCO Crossing the Chasm
Motivated Machine Learning forMotivated Machine Learning for
Water Resource ManagementWater Resource Management
Janusz StarzykSchool of Electrical Engineering and Computer Science, Ohio University, USA
www.ent.ohiou.edu/~starzyk
UNESCO Workshop on Integrated Modeling Approaches to Support Water Resource Decision Making:
Crossing the Chasm
UNESCO Crossing the Chasm
Challenges in Water Management Embodied Intelligence (EI) Embodiment of Mind EI Interaction with Environment How to Motivate a Machine Goal Creation Hierarchy GCS Experiment Promises of EI
To economy To society
Outline
UNESCO Crossing the Chasm
Water management is challenging for various reasons:
Strategies in water management are developed mostly on national level
There is a growing competition between countries for water
Water policy making effects environment and society, health and development, and economy
Growing demands of countries’ populations for water Leads to hydrological nationalism Creates a need to integrate water sciences and policy
making There is an acute need for legitimate scientific data Decision making in water-related health, food and
energy systems are critical to economical development and security
Challenges in Water Management
UNESCO Crossing the Chasm
Decision makers must answer important questions:
How do we make water use sustainable? Who owns the water? What policies, institutional and legal framework
can promote sustainable use of water? How to protect water resources from overuse
and contamination? Water problems became too complex,
interconnected and large to be handled by any one institution or by one group of professionals
It is a challenge to evolve strategies and institutional frameworks for quick policy changes towards an acceptable water use
It is necessary to create assessment and modeling tools to improve policy making and facilitate interaction.
Challenges in Water Management
UNESCO Crossing the Chasm
Why development of integrated modeling to support decision making is important ?
Computerized models were used for many years to support water related decisions.
Models often simplify dynamics of economic, social and environmental interactions and lead to inappropriate policy making and management decisions.
This note proposes models to emerge from interaction with real dynamically changing environments with all of their complexities and societal dependencies.
The main objective is to suggest an integrated modeling framework that may assist with the tasks of water related decision making.
Challenges in Water Management
UNESCO Crossing the Chasm
What are deficiencies of machine created models?
Artificial neural networks, fuzzy logic, and genetic algorithms have all been used to model the hydrological cycle
However, it is still difficult to apply these tools in making real-life water decisions as the related parameters are not explicitly known
What may be needed is a motivated machine learning for characterizing the data and making predictions about their dynamic changes It could support intelligent decision making in dynamically
changing environment It could be used to observe impacts of alternative water
management policies It may consider social, cultural, political, economic and
institutional elements that influence decision making This strategic note presents a goal creation approach in
embodied intelligence (EI) that motivates machine to develop into a useful research toll.
Challenges in Water Management
UNESCO Crossing the Chasm
Embodied intelligence may support decision making:
EI mimics biological intelligent systems, extracting general principles of intelligent behavior and applying them to design intelligent agents
It uses emerging, self-organizing, goal creation (GC) system that motivates embodied intelligence to learn how to efficiently interact with the environment Knowledge is not entered into such systems, but rather
is a result of their successful use in a given environment. This knowledge is validated through active interaction
with the environment. Motivated intelligent systems adapt to unpredictable
and dynamic situations in the environment by learning, which gives them a high degree of autonomy
Learning in such systems is incremental, with continuous prediction of the input associations based on the emerging models - only new information is registered in the memory
Challenges in Water Management
UNESCO Crossing the Chasm
Use the motivated learning scheme to integrate modelling and decision making:
It is suggested to apply ML approach to water management in changing environments where the existing methods fail or work with difficulty. For instance, using classical machine learning to
predict the future for physical processes works only under the assumption that same processes will repeat.
However, if a process changes beyond certain limits, the prediction will not be correct.
GC systems may overcome this difficulty and such systems can be trained to advice humans.
Design concepts, computational mechanisms, and architectural organization of embodied intelligence are presented in this talk
The talk will explain internal motivation mechanism that leads to effective goal oriented learning
In addition, a goal creation mechanism and goal driven learning will be described.
Challenges in Water Management
UNESCO Crossing the Chasm
“…Perhaps the last frontier of science – its ultimate challenge- is to understand the biological basis of consciousness and the mental process by which we perceive, act, learn and remember..” from Principles of Neural Science by E. R. Kandel et al. E. R. Kandel won Nobel Price in 2000 for his work on physiological
basis of memory storage in neurons.
“…The question of intelligence is the last great terrestrial frontier of science...” from Jeff Hawkins On Intelligence. Jeff Hawkins founded the Redwood Neuroscience Institute devoted
to brain research
Intelligence
AI’s holy grailFrom Pattie Maes MIT Media Lab
UNESCO Crossing the Chasm
Traditional AITraditional AI Embodied Intelligence Embodied Intelligence Abstract intelligence
attempt to simulate “highest” human faculties:
– language, discursive reason, mathematics, abstract problem solving
Environment model Condition for problem
solving in abstract way “brain in a vat”
Embodiment knowledge is implicit in the
fact that we have a body– embodiment supports brain
development
Intelligence develops through interaction with environment Situated in environment Environment is its best model
UNESCO Crossing the Chasm
Design principles of intelligent systemsDesign principles of intelligent systemsfrom Rolf Pfeifer “Understanding of Intelligence”, 1999
Interaction with complex environment
cheap design ecological balance redundancy principle parallel, loosely
coupled processes asynchronous sensory-motor
coordination value principle Agent
Drawing by Ciarán O’Leary- Dublin Institute of Technology
UNESCO Crossing the Chasm
Embodied Intelligence Embodied Intelligence
Definition Embodied Intelligence (EI) is a mechanism that learns
how to survive in a hostile environment
– Mechanism: biological, mechanical or virtual agentwith embodied sensors and actuators
– EI acts on environment and perceives its actions– Environment hostility is persistent and stimulates EI to act– Hostility: direct aggression, pain, scarce resources, etc– EI learns so it must have associative self-organizing memory– Knowledge is acquired by EI
UNESCO Crossing the Chasm
Embodiment
Actuators
Sensors
Intelligence core
channel
channel
Embodiment
Sensors
Intelligence core
Environment
channel
channelActuators
Embodiment
Actuators
Sensors
Intelligence core
channel
channel
Embodiment
Sensors
Intelligence core
Environment
channel
channelActuators
Embodiment of a MindEmbodiment of a Mind
Embodiment contains intelligence core and sensory motor interfaces under its control to interact with environment
Necessary for development of intelligence
Not necessarily constant or in the form of a physical body
Boundary transforms modifying brain’s self-determination
UNESCO Crossing the Chasm
Brain learns own body’s dynamic Self-awareness is a result of
identification with own embodiment Embodiment can be extended by
using tools and machines Successful operation is a function
of correct perception of environment and own embodiment
Embodiment of a Mind
UNESCO Crossing the Chasm
INPUT OUTPUT
Simulation or
Real-World System
TaskEnvironment
Agent Architecture
Long-term Memory
Short-term Memory
Reason
ActPerceive
RETRIEVAL LEARNING
EI Interaction with Environment
From Randolph M. Jones, P : www.soartech.com
UNESCO Crossing the Chasm
How to Motivate a MachineHow to Motivate a Machine ? ?
The fundamental question is how to motivate a machine to do anything, in particular to increase its “brain” complexity?
How to motivate it to explore the environment and learn how to effectively work in this environment?
Can a machine that only implements externally given goals be intelligent?If not how these goals can be created?
UNESCO Crossing the Chasm
I suggest that hostility of environment motivates us. It is the pain that moves us. Our intelligence that tries to minimize this pain motivates our actions,
learning and development
We need both the environment hostility and the mechanism that learns how to reduce inflicted by the environment pain
How to Motivate a MachineHow to Motivate a Machine ? ?
In this work I propose, based on the pain, mechanism that motivates the machine to act, learn and develop.
So the pain is good.Without the pain there will be no intelligence. Without the pain there will be no motivation to develop.
UNESCO Crossing the Chasm
Pain-center and Goal CreationPain-center and Goal Creation
Simple Mechanism Creates hierarchy of
values Leads to formulation of
complex goals Reinforcement :
• Pain increase• Pain decrease
Forces exploration
+
-
Environment
Sensor
MotorPain level
Dual pain levelPain increase
Pain decrease
(-)
(+)
Excitation
(-)
(-)
(+)
(+)
Wall-E’s goal is to keep his plants from dying
UNESCO Crossing the Chasm
Primitive Goal CreationPrimitive Goal Creation
- +
Pain
Dry soilPrimitive
level
opentank
sit on garbage
refillfaucet
w. can water
Dual pain
UNESCO Crossing the Chasm
Abstract Goal CreationAbstract Goal Creation The goal is to reduce the primitive pain level Abstract goals are created to reduce abstract pains in order to satisfy the primitive goals Abstract pain center
- +
PainDual pain
+
Dry soil
Abstract pain
“water can” – sensory input
to abstract pain center
Sensory pathway(perception, sense)
Motor pathway(action, reaction)
Primitive Level
Level I
Level IIfaucet
-
w. can
open
water
ActivationStimulationInhibitionReinforcementEchoNeedExpectation
UNESCO Crossing the Chasm
Abstract Goal HierarchyAbstract Goal Hierarchy
A hierarchy of abstract goals is created - they satisfy the lower level goals
ActivationStimulationInhibitionReinforcementEchoNeedExpectation
- +
+
Dry soilPrimitive Level
Level I
Level IIfaucet
-
w. can
open
water
+
Sensory pathway(perception, sense)
Motor pathway(action, reaction)
Level IIItank
-
refill
UNESCO Crossing the Chasm
GCS vs. Reinforcement LearningGCS vs. Reinforcement Learning
Environment
CriticStates
Value Function
Policy
reward
action
Environment
CriticStates
Value Function
Policy
reward
action
RL Actor-critic design Goal creation system
Case study: “How can Wall-E water his plants if the water resources are limited and hard to find?”
Sensorypathway
Motorpathway
GCS
Environment
Pain
States
Gate control
Desired action &state
Action decision
Action
UNESCO Crossing the Chasm
Goal Creation Experiment
Sensory-motor pairs and their effect on the environment
-lake waterfallrain29
lake waterreservoir wateropenpipe22
reservoir waterwater in tankrefilltank15
water in tankwater in canopenfaucet8
water in canmoisturewater the plantwater can1
DECREASESINCREASESMOTORSENSORYPAIR #
UNESCO Crossing the Chasm
Results from GCS schemeResults from GCS scheme
0 100 200 300 400 500 6000
2
4pa
in
Dry soil
0 100 200 300 400 500 6000
1
2
pain
No water in can
0 100 200 300 400 500 6000
1
2
pain
No water in tank
0 100 200 300 400 500 6000
0.5
1
pain
No water in reservoir
0 100 200 300 400 500 6000
2
4
pain
No water in lake
UNESCO Crossing the Chasm
Averaged performance over 10 trials:
GCS:
RL:0 100 200 300 400 500 600
0
0 .5
1
pain
P rim itive p a in
0 100 200 300 400 500 6000
0.5
1
pain
L ac k o f fo o d
0 100 200 300 400 500 6000
0.2
0 .4
pain
L ac k o f m o ne y
0 100 200 300 400 500 6000
0.2
0 .4
pain
L ac k o f b ank s aving s
0 100 200 300 400 500 6000
0.2
0 .4
pain
L ac k o f jo b o p p o rtunity
0 100 200 300 400 500 600-1
0
1
pain
L ac k o f s c ho o l o p p o rtun ity
Machine using GCS learns to control all abstract pains and maintains the primitive pain signal on a low level in
demanding environment conditions.
0 100 200 300 400 500 6000
10
20
30
GCS vs. Reinforcement LearningGCS vs. Reinforcement Learning
UNESCO Crossing the Chasm
Goal Creation Experiment
Action scatters in 5 CGS simulations
0 100 200 300 400 500 6000
5
10
15
20
25
30
35
40Goal Scatter Plot
Go
al ID
Discrete time
UNESCO Crossing the Chasm
Goal Creation Experiment
The average pain signals in 100 CGS simulations
0 100 200 300 400 500 6000
0.5
Primitive pain – dry soil
Pai
n
0 100 200 300 400 500 6000
0.10.2
Lack of water in can
Pai
n
0 100 200 300 400 500 6000
0.10.2
Lack of water in tank
Pai
n
0 100 200 300 400 500 6000
0.10.2
Lack of water in reservoir
Pai
n
0 100 200 300 400 500 6000
0.050.1
Lack of water in lake
Pai
n
Discrete time
UNESCO Crossing the Chasm
Compare RL (TDF) and GCSCompare RL (TDF) and GCS
Mean primitive pain Pp value as a function of the number of iterations.
Dashed lines indicate moment when Pp is getting stable - green line for TDF - blue line for GCS.
UNESCO Crossing the Chasm
Comparison of execution time on log-log scale TD-Falcon green GCS blue
Combined efficiency of GCS 1000 better than TDF
Compare RL (TDF) and GCSCompare RL (TDF) and GCS
Problem solved
Conclusion: embodied intelligence, with motivated learning based on goal creation system, effectively integrates environment modeling and decision making – thus it is poised to cross the chasm
UNESCO Crossing the Chasm
Promises of embodied intelligencePromises of embodied intelligence To society
Advanced use of technology– Robots– Tutors– Intelligent gadgets
Intelligence age follows– Industrial age– Technological age– Information age
Society of minds– Superhuman intelligence– Progress in science– Solution to societies’ ills
To industry Technological development New markets Economical growth
ISAC, a Two-Armed Humanoid RobotVanderbilt University
UNESCO Crossing the Chasm
2002 2010 2020 2030
Biomimetics and Bio-inspired SystemsImpact on Space Transportation, Space Science and Earth Science
Mis
sio
n C
om
ple
xity
Biological Mimicking
Embryonics
Extremophiles
DNA Computing
Brain-like computing
Self Assembled Array
Artificial nanoporehigh resolution
Mars in situlife detector
Sensor Web
Skin and Bone
Self healing structureand thermal protection
systems
Biologically inspired aero-space systems
Space Transportation
Memristors
Biological nanoporelow resolution
UNESCO Crossing the Chasm
Sounds like science fictionSounds like science fiction
If you’re trying to look far ahead, and what you see seems like science fiction, it might be wrong.
But if it doesn’t seem like science fiction, it’s definitely wrong.
From presentation by Foresight Institute
UNESCO Crossing the Chasm
Questions?Questions?