CS 416Artificial Intelligence

Lecture 25Lecture 25

AI ApplicationsAI Applications

Lecture 25Lecture 25

AI ApplicationsAI Applications

Final Exam

Saturday, May 6Saturday, May 6th th at 7:00 pmat 7:00 pm

Should only conflict with French finalsShould only conflict with French finals

It will not be cumulativeIt will not be cumulative

Saturday, May 6Saturday, May 6th th at 7:00 pmat 7:00 pm

Should only conflict with French finalsShould only conflict with French finals

It will not be cumulativeIt will not be cumulative

NN Assignment

Due May 2Due May 2ndndDue May 2Due May 2ndnd

Some general artificial neural network (ANN) info

• The entire network is a function g( inputs ) = outputsThe entire network is a function g( inputs ) = outputs

– These functions frequently have sigmoids in themThese functions frequently have sigmoids in them

– These functions are frequently differentiableThese functions are frequently differentiable

– These functions have coefficients (weights)These functions have coefficients (weights)

• Backpropagation networks are simply ways to tune the Backpropagation networks are simply ways to tune the coefficients of a function so it produces desired outputcoefficients of a function so it produces desired output

• The entire network is a function g( inputs ) = outputsThe entire network is a function g( inputs ) = outputs

– These functions frequently have sigmoids in themThese functions frequently have sigmoids in them

– These functions are frequently differentiableThese functions are frequently differentiable

– These functions have coefficients (weights)These functions have coefficients (weights)

• Backpropagation networks are simply ways to tune the Backpropagation networks are simply ways to tune the coefficients of a function so it produces desired outputcoefficients of a function so it produces desired output

Function approximation

Consider fitting a line to dataConsider fitting a line to data

• Coefficients: slope and y-interceptCoefficients: slope and y-intercept

• Training data: some samplesTraining data: some samples

• Use least-squares fitUse least-squares fit

This is what an ANN doesThis is what an ANN does

Consider fitting a line to dataConsider fitting a line to data

• Coefficients: slope and y-interceptCoefficients: slope and y-intercept

• Training data: some samplesTraining data: some samples

• Use least-squares fitUse least-squares fit

This is what an ANN doesThis is what an ANN does

x

y

Function approximation

A function of two inputs…A function of two inputs…• Fit a smoothFit a smooth

curve to thecurve to theavailableavailabledatadata

– QuadraticQuadratic

– CubicCubic

– nnthth-order-order

– ANN!ANN!

A function of two inputs…A function of two inputs…• Fit a smoothFit a smooth

curve to thecurve to theavailableavailabledatadata

– QuadraticQuadratic

– CubicCubic

– nnthth-order-order

– ANN!ANN!

Curve fitting

• A neural network should be able to generate the A neural network should be able to generate the input/output pairs from the training datainput/output pairs from the training data

• You’d like for it to be smooth (and well-behaved) in the You’d like for it to be smooth (and well-behaved) in the voids between the training datavoids between the training data

• There are risks of over fitting the dataThere are risks of over fitting the data

• A neural network should be able to generate the A neural network should be able to generate the input/output pairs from the training datainput/output pairs from the training data

• You’d like for it to be smooth (and well-behaved) in the You’d like for it to be smooth (and well-behaved) in the voids between the training datavoids between the training data

• There are risks of over fitting the dataThere are risks of over fitting the data

When using ANNs

• Sometimes the output layer feeds back into the input Sometimes the output layer feeds back into the input layer – recurrent neural networkslayer – recurrent neural networks

• The backpropagation will tune the weightsThe backpropagation will tune the weights

• You determine the topologyYou determine the topology

– Different topologies have different training outcomes Different topologies have different training outcomes (consider overfitting)(consider overfitting)

– Sometimes a genetic algorithm is used to explore Sometimes a genetic algorithm is used to explore the space of neural network topologiesthe space of neural network topologies

• Sometimes the output layer feeds back into the input Sometimes the output layer feeds back into the input layer – recurrent neural networkslayer – recurrent neural networks

• The backpropagation will tune the weightsThe backpropagation will tune the weights

• You determine the topologyYou determine the topology

– Different topologies have different training outcomes Different topologies have different training outcomes (consider overfitting)(consider overfitting)

– Sometimes a genetic algorithm is used to explore Sometimes a genetic algorithm is used to explore the space of neural network topologiesthe space of neural network topologies

What is the Purpose of NN?

To create an Artificial Intelligence, orTo create an Artificial Intelligence, or

• Although not an invalid purpose, many people in the AI Although not an invalid purpose, many people in the AI community think neural networks do not provide anything community think neural networks do not provide anything that cannot be obtained through other techniquesthat cannot be obtained through other techniques

– It is hard to unravel the “intelligence” behind why the It is hard to unravel the “intelligence” behind why the ANN worksANN works

To study how the human brain works?To study how the human brain works?

• Ironically, those studying neural networks with this in mind Ironically, those studying neural networks with this in mind are more likely to contribute to the previous purposeare more likely to contribute to the previous purpose

To create an Artificial Intelligence, orTo create an Artificial Intelligence, or

• Although not an invalid purpose, many people in the AI Although not an invalid purpose, many people in the AI community think neural networks do not provide anything community think neural networks do not provide anything that cannot be obtained through other techniquesthat cannot be obtained through other techniques

– It is hard to unravel the “intelligence” behind why the It is hard to unravel the “intelligence” behind why the ANN worksANN works

To study how the human brain works?To study how the human brain works?

• Ironically, those studying neural networks with this in mind Ironically, those studying neural networks with this in mind are more likely to contribute to the previous purposeare more likely to contribute to the previous purpose

Some Brain Facts

• Contains ~100,000,000,000 neuronsContains ~100,000,000,000 neurons

• Hippocampus CA3 region contains Hippocampus CA3 region contains ~3,000,000 neurons~3,000,000 neurons

• Each neuron is connected to ~10,000 other Each neuron is connected to ~10,000 other neuronsneurons

• ~ (10~ (101515)(10)(101515) connections!) connections!

• Consumes ~20-30% of the body’s energyConsumes ~20-30% of the body’s energy

• Contains about 2% of the body’s massContains about 2% of the body’s mass

• Contains ~100,000,000,000 neuronsContains ~100,000,000,000 neurons

• Hippocampus CA3 region contains Hippocampus CA3 region contains ~3,000,000 neurons~3,000,000 neurons

• Each neuron is connected to ~10,000 other Each neuron is connected to ~10,000 other neuronsneurons

• ~ (10~ (101515)(10)(101515) connections!) connections!

• Consumes ~20-30% of the body’s energyConsumes ~20-30% of the body’s energy

• Contains about 2% of the body’s massContains about 2% of the body’s mass

Karl Sims

Evolved Virtual Creatures (SIGGRAPH 1994)Evolved Virtual Creatures (SIGGRAPH 1994)

Evolving 3D Morphology and Behavior by Evolving 3D Morphology and Behavior by Competition (ALIFECompetition (ALIFE1994)1994)

Evolved Virtual Creatures (SIGGRAPH 1994)Evolved Virtual Creatures (SIGGRAPH 1994)

Evolving 3D Morphology and Behavior by Evolving 3D Morphology and Behavior by Competition (ALIFECompetition (ALIFE1994)1994)

MotivationMotivation

Automatically generate physically Automatically generate physically simulated characters without fully simulated characters without fully understanding the procedures used to understanding the procedures used to build and control thembuild and control them

• MorphologyMorphology

• ControlControl

Automatically generate physically Automatically generate physically simulated characters without fully simulated characters without fully understanding the procedures used to understanding the procedures used to build and control thembuild and control them

• MorphologyMorphology

• ControlControl

Creating animation from scratch

Typical ScenarioTypical Scenario

• 20 seconds of animation20 seconds of animation

• 17 controlled DOFs17 controlled DOFs

• (20 * 30 * 17) = 10,200 torque values to determine(20 * 30 * 17) = 10,200 torque values to determine

• No approximations to best answer are knownNo approximations to best answer are known

– Spacetime constraints would use initial guess to find Spacetime constraints would use initial guess to find local maximumlocal maximum

Typical ScenarioTypical Scenario

• 20 seconds of animation20 seconds of animation

• 17 controlled DOFs17 controlled DOFs

• (20 * 30 * 17) = 10,200 torque values to determine(20 * 30 * 17) = 10,200 torque values to determine

• No approximations to best answer are knownNo approximations to best answer are known

– Spacetime constraints would use initial guess to find Spacetime constraints would use initial guess to find local maximumlocal maximum

Creating animation from scratch

Searching for 10,200 unknownsSearching for 10,200 unknowns

• We cannot prescribe initial values with confidenceWe cannot prescribe initial values with confidence

• We can prescribe some behaviors and principles we We can prescribe some behaviors and principles we think the motion will demonstratethink the motion will demonstrate

– Insight: the movements at some DOFs are functions Insight: the movements at some DOFs are functions of movements at other DOFsof movements at other DOFs

Sun and Metaxas – Automating Sun and Metaxas – Automating Gait Generation, SIGGRAPH 01Gait Generation, SIGGRAPH 01

Searching for 10,200 unknownsSearching for 10,200 unknowns

• We cannot prescribe initial values with confidenceWe cannot prescribe initial values with confidence

• We can prescribe some behaviors and principles we We can prescribe some behaviors and principles we think the motion will demonstratethink the motion will demonstrate

– Insight: the movements at some DOFs are functions Insight: the movements at some DOFs are functions of movements at other DOFsof movements at other DOFs

Sun and Metaxas – Automating Sun and Metaxas – Automating Gait Generation, SIGGRAPH 01Gait Generation, SIGGRAPH 01

So we’re not really working from scratch

Searching for 10,200 unknownsSearching for 10,200 unknowns

• How to incorporate our motion heuristics into search?How to incorporate our motion heuristics into search?

– Simulated annealing?Simulated annealing?

– Neural networks?Neural networks?

– Genetic algorithms?Genetic algorithms?

Searching for 10,200 unknownsSearching for 10,200 unknowns

• How to incorporate our motion heuristics into search?How to incorporate our motion heuristics into search?

– Simulated annealing?Simulated annealing?

– Neural networks?Neural networks?

– Genetic algorithms?Genetic algorithms?

But we don’t know the morphology

The previous example assumed a known The previous example assumed a known morphologymorphology

• We’re searching for best architecture AND best control We’re searching for best architecture AND best control simultaneously!simultaneously!

Our search space just grew to an unknown Our search space just grew to an unknown degree!degree!

The previous example assumed a known The previous example assumed a known morphologymorphology

• We’re searching for best architecture AND best control We’re searching for best architecture AND best control simultaneously!simultaneously!

Our search space just grew to an unknown Our search space just grew to an unknown degree!degree!

Imitate the processes of evolution and natural Imitate the processes of evolution and natural selection in biological world to selection in biological world to generate/evolve the best solutiongenerate/evolve the best solution

Imitate the processes of evolution and natural Imitate the processes of evolution and natural selection in biological world to selection in biological world to generate/evolve the best solutiongenerate/evolve the best solution

Genetic Algorithms

(not to scale)

Genetic Algorithms

Inject user knowledgeInject user knowledge

• Objective function Objective function

• Genetic representationGenetic representation

– Let’s search for collection of jointsLet’s search for collection of joints

– Let’s not search for n * 30 * 20 unknown joint Let’s not search for n * 30 * 20 unknown joint torquestorques

– Instead, search for n equations that are logical Instead, search for n equations that are logical combinations of virtual sensors of creature statecombinations of virtual sensors of creature state

– These equations are reused for all timestepsThese equations are reused for all timesteps

Inject user knowledgeInject user knowledge

• Objective function Objective function

• Genetic representationGenetic representation

– Let’s search for collection of jointsLet’s search for collection of joints

– Let’s not search for n * 30 * 20 unknown joint Let’s not search for n * 30 * 20 unknown joint torquestorques

– Instead, search for n equations that are logical Instead, search for n equations that are logical combinations of virtual sensors of creature statecombinations of virtual sensors of creature state

– These equations are reused for all timestepsThese equations are reused for all timesteps

Sensibility of GA for this problem

Motion principles described as heuristicsMotion principles described as heuristics• Minimize angular disturbances by swinging legs in Minimize angular disturbances by swinging legs in

opposite directionsopposite directions

• Maximize balance by keeping center of mass above Maximize balance by keeping center of mass above platform of supportplatform of support

• Kinematics of arm are similar to kinematics of legKinematics of arm are similar to kinematics of leg

So let’s find those heuristics and permit them So let’s find those heuristics and permit them to be shared across limbsto be shared across limbs

Motion principles described as heuristicsMotion principles described as heuristics• Minimize angular disturbances by swinging legs in Minimize angular disturbances by swinging legs in

opposite directionsopposite directions

• Maximize balance by keeping center of mass above Maximize balance by keeping center of mass above platform of supportplatform of support

• Kinematics of arm are similar to kinematics of legKinematics of arm are similar to kinematics of leg

So let’s find those heuristics and permit them So let’s find those heuristics and permit them to be shared across limbsto be shared across limbs

Genetic Representation

Defined by creature morphology and Defined by creature morphology and neurologyneurology

Morphology:Morphology: “The form and structure of an “The form and structure of an organism or one of its parts”organism or one of its parts”

Neurology Neurology (Nervous system): “The system of (Nervous system): “The system of cells, tissues, and organs that regulates the cells, tissues, and organs that regulates the body's responses to internal and external body's responses to internal and external stimuli”stimuli”

Defined by creature morphology and Defined by creature morphology and neurologyneurology

Morphology:Morphology: “The form and structure of an “The form and structure of an organism or one of its parts”organism or one of its parts”

Neurology Neurology (Nervous system): “The system of (Nervous system): “The system of cells, tissues, and organs that regulates the cells, tissues, and organs that regulates the body's responses to internal and external body's responses to internal and external stimuli”stimuli”

Creature MorphologyCreature Morphology

Articulated three-Articulated three-dimension rigid body dimension rigid body partsparts

Represented as a directed Represented as a directed graph of nodes and graph of nodes and connectionsconnections

Each node describes one Each node describes one body partbody part

• Many parameters to a nodeMany parameters to a node

Articulated three-Articulated three-dimension rigid body dimension rigid body partsparts

Represented as a directed Represented as a directed graph of nodes and graph of nodes and connectionsconnections

Each node describes one Each node describes one body partbody part

• Many parameters to a nodeMany parameters to a node

Creature Neurology

Comprised of:Comprised of:

• Sensors Sensors

• NeuronsNeurons

• EffectorsEffectors

Comprised of:Comprised of:

• Sensors Sensors

• NeuronsNeurons

• EffectorsEffectors

SensorsSensors

Each sensor is at a different part of the bodyEach sensor is at a different part of the body

• It measuresIt measures

– That body partThat body part

– The world relative to that body partThe world relative to that body part

• TypesTypes

– Joint angle sensorJoint angle sensor

– Contact sensorsContact sensors

– Photo sensorsPhoto sensors

Each sensor is at a different part of the bodyEach sensor is at a different part of the body

• It measuresIt measures

– That body partThat body part

– The world relative to that body partThe world relative to that body part

• TypesTypes

– Joint angle sensorJoint angle sensor

– Contact sensorsContact sensors

– Photo sensorsPhoto sensors

NeuronsNeurons

Virtual brain: Function based on sensor Virtual brain: Function based on sensor input input

Provides output effector valuesProvides output effector values

Virtual brain: Function based on sensor Virtual brain: Function based on sensor input input

Provides output effector valuesProvides output effector values

EffectorsEffectors

Each effector controls a degree of Each effector controls a degree of freedom in a jointfreedom in a joint

Each effector’s value is applied as forces Each effector’s value is applied as forces or torquesor torques

Only exert simulated muscle forceOnly exert simulated muscle force

Connected to neuron or sensorConnected to neuron or sensor

Each effector controls a degree of Each effector controls a degree of freedom in a jointfreedom in a joint

Each effector’s value is applied as forces Each effector’s value is applied as forces or torquesor torques

Only exert simulated muscle forceOnly exert simulated muscle force

Connected to neuron or sensorConnected to neuron or sensor

Neural SystemNeural System

Local neural system:Local neural system:

• Neural system generated along with morphological systemNeural system generated along with morphological system

• Connection allowed between local neural systemsConnection allowed between local neural systems

Global neural system:Global neural system:

• Neurons not associated with a specific nodeNeurons not associated with a specific node

– Useful heuristic for creating global timing mechanismsUseful heuristic for creating global timing mechanisms

• Can be connected with local neural systemsCan be connected with local neural systems

Local neural system:Local neural system:

• Neural system generated along with morphological systemNeural system generated along with morphological system

• Connection allowed between local neural systemsConnection allowed between local neural systems

Global neural system:Global neural system:

• Neurons not associated with a specific nodeNeurons not associated with a specific node

– Useful heuristic for creating global timing mechanismsUseful heuristic for creating global timing mechanisms

• Can be connected with local neural systemsCan be connected with local neural systems

Neural System

• Sum, product, divide, Sum, product, divide, threshold, >, sign, min, abs, if, threshold, >, sign, min, abs, if, interpolate, sine, cosine, atan, interpolate, sine, cosine, atan, log, exponent, sigmoid, log, exponent, sigmoid, integrate, differentiate, integrate, differentiate, smooth, memory, wave, smooth, memory, wave, sawtoothsawtooth

• Sum, product, divide, Sum, product, divide, threshold, >, sign, min, abs, if, threshold, >, sign, min, abs, if, interpolate, sine, cosine, atan, interpolate, sine, cosine, atan, log, exponent, sigmoid, log, exponent, sigmoid, integrate, differentiate, integrate, differentiate, smooth, memory, wave, smooth, memory, wave, sawtoothsawtooth

Physical Simulation

• Featherstone’s articulated body methodFeatherstone’s articulated body method

• Runge-Kutta-FehlbergRunge-Kutta-Fehlberg

– Adaptive timestep typically 1-5 steps per 1/30 secondAdaptive timestep typically 1-5 steps per 1/30 second

• Bounding boxes to reduce collision checksBounding boxes to reduce collision checks

– Connected pairs can interpenetrate but cannot rotate Connected pairs can interpenetrate but cannot rotate entirely through one anotherentirely through one another

• Impulses and springs used for collision correctionsImpulses and springs used for collision corrections

• Viscosity used for underwaterViscosity used for underwater

• Featherstone’s articulated body methodFeatherstone’s articulated body method

• Runge-Kutta-FehlbergRunge-Kutta-Fehlberg

– Adaptive timestep typically 1-5 steps per 1/30 secondAdaptive timestep typically 1-5 steps per 1/30 second

• Bounding boxes to reduce collision checksBounding boxes to reduce collision checks

– Connected pairs can interpenetrate but cannot rotate Connected pairs can interpenetrate but cannot rotate entirely through one anotherentirely through one another

• Impulses and springs used for collision correctionsImpulses and springs used for collision corrections

• Viscosity used for underwaterViscosity used for underwater

Genetic Operators: Mutation Alter internal node parameters

Dimensions- determine the physical shape of the Dimensions- determine the physical shape of the partpart

Joint type- number of DOF for each joint and Joint type- number of DOF for each joint and movement allowed for each DOFmovement allowed for each DOF

Joint limits- point where spring forces will be Joint limits- point where spring forces will be exerted for each DOFexerted for each DOF

Recursive limitRecursive limit

Set of local neuronsSet of local neurons

Set of connections to other nodesSet of connections to other nodes• Position, orientation, scale and reflection relative to Position, orientation, scale and reflection relative to

parent parent

Dimensions- determine the physical shape of the Dimensions- determine the physical shape of the partpart

Joint type- number of DOF for each joint and Joint type- number of DOF for each joint and movement allowed for each DOFmovement allowed for each DOF

Joint limits- point where spring forces will be Joint limits- point where spring forces will be exerted for each DOFexerted for each DOF

Recursive limitRecursive limit

Set of local neuronsSet of local neurons

Set of connections to other nodesSet of connections to other nodes• Position, orientation, scale and reflection relative to Position, orientation, scale and reflection relative to

parent parent

Genetic Operators: Mutation Add a new node at random

Leg Segment

BodySegment

Head Segment

Genetic Operators: MutationParameter of each connection can change

Leg Segment

BodySegment

9045

Genetic Operators: MutationAdd/remove connections

Leg Segment

BodySegment

Head Segment

Genetic Operators: Mutation Delete unconnected nodes

Leg Segment

BodySegment

Head Segment

Genetic Operators: Crossover

Genetic Operators: Grafting

StartStart

Generate random population of Generate random population of nn

suitable solutions for the problemsuitable solutions for the problem • Random generation of nodes and connectionsRandom generation of nodes and connections

• Use existing genotype to seed functionUse existing genotype to seed function

• Create genotype manually to seed functionCreate genotype manually to seed function

Generate random population of Generate random population of nn

suitable solutions for the problemsuitable solutions for the problem • Random generation of nodes and connectionsRandom generation of nodes and connections

• Use existing genotype to seed functionUse existing genotype to seed function

• Create genotype manually to seed functionCreate genotype manually to seed function

Fitness

Swimming, walking, jumping Swimming, walking, jumping • Distance traveled by COM/unit of timeDistance traveled by COM/unit of time

FollowingFollowing• Speed at which creature moves towards targetSpeed at which creature moves towards target

CompetitionCompetition• Pairs of individuals compete for control over a cubePairs of individuals compete for control over a cube• Final distance of the creature from cube and Final distance of the creature from cube and

opponents final distance from cubeopponents final distance from cube

Swimming, walking, jumping Swimming, walking, jumping • Distance traveled by COM/unit of timeDistance traveled by COM/unit of time

FollowingFollowing• Speed at which creature moves towards targetSpeed at which creature moves towards target

CompetitionCompetition• Pairs of individuals compete for control over a cubePairs of individuals compete for control over a cube• Final distance of the creature from cube and Final distance of the creature from cube and

opponents final distance from cubeopponents final distance from cube

New population

• Survival ratio: 1/5 of 300Survival ratio: 1/5 of 300

• 40% mutation, 30% crossover, 30% 40% mutation, 30% crossover, 30% graftinggrafting

• Children from crossover and grafting Children from crossover and grafting are sometimes mutatedare sometimes mutated

• Survival ratio: 1/5 of 300Survival ratio: 1/5 of 300

• 40% mutation, 30% crossover, 30% 40% mutation, 30% crossover, 30% graftinggrafting

• Children from crossover and grafting Children from crossover and grafting are sometimes mutatedare sometimes mutated

ResultsResults

Different runs converged on characters Different runs converged on characters with different evolved characteristicswith different evolved characteristics

Specify a User Preference using the Specify a User Preference using the AlgorithmAlgorithm

• User can specify a preference over multiple runs User can specify a preference over multiple runs of the simulationof the simulation

• User can perform the natural selectionUser can perform the natural selection

Different runs converged on characters Different runs converged on characters with different evolved characteristicswith different evolved characteristics

Specify a User Preference using the Specify a User Preference using the AlgorithmAlgorithm

• User can specify a preference over multiple runs User can specify a preference over multiple runs of the simulationof the simulation

• User can perform the natural selectionUser can perform the natural selection

Results

VideoVideoVideoVideo

AdvantagesAdvantages

Generate characters without specification Generate characters without specification or knowledge about exactly how they or knowledge about exactly how they workwork

Less likely to get stuck on a local solution Less likely to get stuck on a local solution since we are searching on an encodingsince we are searching on an encoding

Easy to parallelizeEasy to parallelize

Generate characters without specification Generate characters without specification or knowledge about exactly how they or knowledge about exactly how they workwork

Less likely to get stuck on a local solution Less likely to get stuck on a local solution since we are searching on an encodingsince we are searching on an encoding

Easy to parallelizeEasy to parallelize

DisadvantagesDisadvantages

The characters could produce incorrect behavior The characters could produce incorrect behavior that could go undetected because of the lack that could go undetected because of the lack of understanding of the underlying algorithmof understanding of the underlying algorithm

Process is not very reusableProcess is not very reusable

• Can reuse some of the components of the implementation, Can reuse some of the components of the implementation, for example, the neuron languagefor example, the neuron language

Genetic Algorithms sometimes require a lot of Genetic Algorithms sometimes require a lot of tweaking to work righttweaking to work right

The characters could produce incorrect behavior The characters could produce incorrect behavior that could go undetected because of the lack that could go undetected because of the lack of understanding of the underlying algorithmof understanding of the underlying algorithm

Process is not very reusableProcess is not very reusable

• Can reuse some of the components of the implementation, Can reuse some of the components of the implementation, for example, the neuron languagefor example, the neuron language

Genetic Algorithms sometimes require a lot of Genetic Algorithms sometimes require a lot of tweaking to work righttweaking to work right

DisadvantagesDisadvantages

Difficult to obtain ideal size for the neuron Difficult to obtain ideal size for the neuron language, node types and parameters (joint language, node types and parameters (joint types and parameters and connection types and parameters and connection parameters), sensor and effector types parameters), sensor and effector types

i.e. Difficult to obtain best size for the search i.e. Difficult to obtain best size for the search spacespace • If the size is too small, then there aren't enough variations If the size is too small, then there aren't enough variations

in the populationin the population

• If the size is too large, then changes in isolated parts of If the size is too large, then changes in isolated parts of the creature do not have a big effect on the behavior or the creature do not have a big effect on the behavior or structurestructure

Difficult to obtain ideal size for the neuron Difficult to obtain ideal size for the neuron language, node types and parameters (joint language, node types and parameters (joint types and parameters and connection types and parameters and connection parameters), sensor and effector types parameters), sensor and effector types

i.e. Difficult to obtain best size for the search i.e. Difficult to obtain best size for the search spacespace • If the size is too small, then there aren't enough variations If the size is too small, then there aren't enough variations

in the populationin the population

• If the size is too large, then changes in isolated parts of If the size is too large, then changes in isolated parts of the creature do not have a big effect on the behavior or the creature do not have a big effect on the behavior or structurestructure

NeuroAnimator

R. Grzeszczuk et al.  SIGGRAPH 1998

Motivation

Physical SimulationPhysical Simulation

• Produces very realistic outputProduces very realistic output

• Highly automatedHighly automated

• Computationally expensiveComputationally expensive

• Difficult to control (author controllers)Difficult to control (author controllers)

Physical SimulationPhysical Simulation

• Produces very realistic outputProduces very realistic output

• Highly automatedHighly automated

• Computationally expensiveComputationally expensive

• Difficult to control (author controllers)Difficult to control (author controllers)

Motivation

What is it about simulations?What is it about simulations?

• (Differential) equations of motion are specified(Differential) equations of motion are specified

– The equations may be brittleThe equations may be brittle

• Small changes in state at time Small changes in state at time (t)(t) may result in may result in large changes at time large changes at time (t+1)(t+1)

• Integration is required and error is possibleIntegration is required and error is possible

• Time steps may be small Time steps may be small

What is it about simulations?What is it about simulations?

• (Differential) equations of motion are specified(Differential) equations of motion are specified

– The equations may be brittleThe equations may be brittle

• Small changes in state at time Small changes in state at time (t)(t) may result in may result in large changes at time large changes at time (t+1)(t+1)

• Integration is required and error is possibleIntegration is required and error is possible

• Time steps may be small Time steps may be small

Emulation

What if we could replace the “simulation”What if we could replace the “simulation”

• Replacing the equations of motion?Replacing the equations of motion?

• Replacing the numerical integration?Replacing the numerical integration?

• Both?Both?

What if we could replace the “simulation”What if we could replace the “simulation”

• Replacing the equations of motion?Replacing the equations of motion?

• Replacing the numerical integration?Replacing the numerical integration?

• Both?Both?

Emulation

Replacing simulation with neural networkReplacing simulation with neural network

• What is gained?What is gained?

• What is lost?What is lost?

Replacing simulation with neural networkReplacing simulation with neural network

• What is gained?What is gained?

• What is lost?What is lost?

Reminder about neural networks

They are a universal function approximatorThey are a universal function approximator

• What function are we approximating?What function are we approximating?

– Equations of motionEquations of motion

• What do they look like?What do they look like?

– f(sf(stt) = s) = st+1t+1

• Can neural networks learn this?Can neural networks learn this?

They are a universal function approximatorThey are a universal function approximator

• What function are we approximating?What function are we approximating?

– Equations of motionEquations of motion

• What do they look like?What do they look like?

– f(sf(stt) = s) = st+1t+1

• Can neural networks learn this?Can neural networks learn this?

Artificial Neurons

““learning”learning”

• Transfer functions, weights, topologiesTransfer functions, weights, topologies

““learning”learning”

• Transfer functions, weights, topologiesTransfer functions, weights, topologies

Artificial Neurons

““learning”learning”

• Training requires dataTraining requires data

– Underfitting – have lots of neuronsUnderfitting – have lots of neurons

– Overfitting - 8-10 times as many examples are used Overfitting - 8-10 times as many examples are used as there are weights in the networkas there are weights in the network

• Backprop to tune the weightsBackprop to tune the weights

““learning”learning”

• Training requires dataTraining requires data

– Underfitting – have lots of neuronsUnderfitting – have lots of neurons

– Overfitting - 8-10 times as many examples are used Overfitting - 8-10 times as many examples are used as there are weights in the networkas there are weights in the network

• Backprop to tune the weightsBackprop to tune the weights

The emulation

A simulation A simulation neural network neural network

• sst+t+tt = sim (s = sim (stt, u, utt, f, ftt))

• sst+t+tt = N = N (s (stt, u, utt, f, ftt) )

– Timestep of ANN is (much) largerTimestep of ANN is (much) larger

A simulation A simulation neural network neural network

• sst+t+tt = sim (s = sim (stt, u, utt, f, ftt))

• sst+t+tt = N = N (s (stt, u, utt, f, ftt) )

– Timestep of ANN is (much) largerTimestep of ANN is (much) larger

Some initial concerns

Basic ANN needs some changes in setupBasic ANN needs some changes in setup

• Mapping from sMapping from stt to s to st+t+tt is nonlinear and has a great is nonlinear and has a great

rangerange

– Position and velocity can vary from +/- infPosition and velocity can vary from +/- inf

– Sigmoid can capture nonlinearities, but its range is Sigmoid can capture nonlinearities, but its range is limitedlimited

• Could use many sigmoids and shift/scale them to Could use many sigmoids and shift/scale them to cover the spacecover the space

Basic ANN needs some changes in setupBasic ANN needs some changes in setup

• Mapping from sMapping from stt to s to st+t+tt is nonlinear and has a great is nonlinear and has a great

rangerange

– Position and velocity can vary from +/- infPosition and velocity can vary from +/- inf

– Sigmoid can capture nonlinearities, but its range is Sigmoid can capture nonlinearities, but its range is limitedlimited

• Could use many sigmoids and shift/scale them to Could use many sigmoids and shift/scale them to cover the spacecover the space

Some initial concerns

Changing basic ANN setupChanging basic ANN setup

• Learn Learn sstt instead of s instead of st+1t+1

• sst+t+tt = s = stt + + sstt

• Some variables are invariant to world positionSome variables are invariant to world position

– Normalize those variables to always be local Normalize those variables to always be local coordinatescoordinates

Changing basic ANN setupChanging basic ANN setup

• Learn Learn sstt instead of s instead of st+1t+1

• sst+t+tt = s = stt + + sstt

• Some variables are invariant to world positionSome variables are invariant to world position

– Normalize those variables to always be local Normalize those variables to always be local coordinatescoordinates

Some initial concerns

Changing basic ANN setupChanging basic ANN setup

• NormalizeNormalize

– Variables with larger magnitudes have bigger impact Variables with larger magnitudes have bigger impact on ANN output even though they may have smaller on ANN output even though they may have smaller impacts on simulation behaviorimpacts on simulation behavior

– Transform all variables to zero-mean, SD = 1Transform all variables to zero-mean, SD = 1

Changing basic ANN setupChanging basic ANN setup

• NormalizeNormalize

– Variables with larger magnitudes have bigger impact Variables with larger magnitudes have bigger impact on ANN output even though they may have smaller on ANN output even though they may have smaller impacts on simulation behaviorimpacts on simulation behavior

– Transform all variables to zero-mean, SD = 1Transform all variables to zero-mean, SD = 1

Some initial concerns

Learning with monolithic networks is toughLearning with monolithic networks is tough

• Should the neurons affecting neck be influenced by Should the neurons affecting neck be influenced by lower leg?lower leg?

• Can we partition the variables Can we partition the variables a prioria priori and learn and learn multiple ANNs?multiple ANNs?

Learning with monolithic networks is toughLearning with monolithic networks is tough

• Should the neurons affecting neck be influenced by Should the neurons affecting neck be influenced by lower leg?lower leg?

• Can we partition the variables Can we partition the variables a prioria priori and learn and learn multiple ANNs?multiple ANNs?

Some initial concerns

Sampling simulation statesSampling simulation states

• Uniform sampling across all inputs is desirable?Uniform sampling across all inputs is desirable?

– It’s too complex anywaysIt’s too complex anyways

– Focus on what is typically encounteredFocus on what is typically encountered

– Grab data points Grab data points in situ in situ while simulation is runningwhile simulation is running

Sampling simulation statesSampling simulation states

• Uniform sampling across all inputs is desirable?Uniform sampling across all inputs is desirable?

– It’s too complex anywaysIt’s too complex anyways

– Focus on what is typically encounteredFocus on what is typically encountered

– Grab data points Grab data points in situ in situ while simulation is runningwhile simulation is running

Training

Examples later will demonstrate the ability to Examples later will demonstrate the ability to learn the dynamicslearn the dynamicsExamples later will demonstrate the ability to Examples later will demonstrate the ability to learn the dynamicslearn the dynamics

Control

Hard to control physical simulationsHard to control physical simulations

• Typically trial-and-errorTypically trial-and-error

– Markov Chain Monte Carlo (MCMC) by ChenneyMarkov Chain Monte Carlo (MCMC) by Chenney

– Sims’ creaturesSims’ creatures

– Simulated annealingSimulated annealing

• Having gradient is usefulHaving gradient is useful

Hard to control physical simulationsHard to control physical simulations

• Typically trial-and-errorTypically trial-and-error

– Markov Chain Monte Carlo (MCMC) by ChenneyMarkov Chain Monte Carlo (MCMC) by Chenney

– Sims’ creaturesSims’ creatures

– Simulated annealingSimulated annealing

• Having gradient is usefulHaving gradient is useful

Control

ANNs are differentiableANNs are differentiable

• Find best control decisions for simulationFind best control decisions for simulation

ANNs are differentiableANNs are differentiable

• Find best control decisions for simulationFind best control decisions for simulation

Performance

• Neural network: O(nNeural network: O(n22), n = state parameter size), n = state parameter size

• Simulation: O (nSimulation: O (n33))

• Delta t for ANN is 50 times larger than simulationDelta t for ANN is 50 times larger than simulation

• Neural network: O(nNeural network: O(n22), n = state parameter size), n = state parameter size

• Simulation: O (nSimulation: O (n33))

• Delta t for ANN is 50 times larger than simulationDelta t for ANN is 50 times larger than simulation

Computation Time

Error

Takeaway

• ANNs can learn simulation behaviorsANNs can learn simulation behaviors

– Normalization, hierarchical networks, localization Normalization, hierarchical networks, localization were all requiredwere all required

– Training was limited to space of typical behaviorTraining was limited to space of typical behavior

• Control is easier to learn with ANNs because they are Control is easier to learn with ANNs because they are analytically differentiableanalytically differentiable

• ANNs can learn simulation behaviorsANNs can learn simulation behaviors

– Normalization, hierarchical networks, localization Normalization, hierarchical networks, localization were all requiredwere all required

– Training was limited to space of typical behaviorTraining was limited to space of typical behavior

• Control is easier to learn with ANNs because they are Control is easier to learn with ANNs because they are analytically differentiableanalytically differentiable