DDDAMS-based Planning and
Control
Young-Jun Son ([email protected])
Systems and Industrial Engineering
University of Arizona
Co-PIs: Jian Liu, Jyh-Ming Lien (George Mason)
Students: S. Minaeian, S. Lee, Y. Yuan, A. Khaleghi, D. Xu, Z. Wang, M. Li, N. Celik, K. Vasudevan
Sponsors:
AFOSR: FA9550-12-1-0238 (DDDAS); FA9550-17-
1-0075 (DDDAS); NSF-CNS: 0540212
Program Manager: Dr. Frederica Darema
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Theme and Topic
• (Workshop) Towards an Ecosystem of Simulation Models and Data
• (Topic) Online, data-driven simulations– Internet/Integration of things (data, model, embedded
algorithms, and hardware/human)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Simulation-Based Planning and Control
• Simulation-based planning and real-time control (shop floor control) (since late 1990’s)
• Extension to top level supply chain activities via DDDAMS
– Via an NSF project (2005~2007)
• UAV/UGV control via DDDAMS
– Via two AFOSR projects (2012~current)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Simulation-based Planning and Control
Y. Son, S. Joshi, R. Wysk, and J. Smith, Simulation Based Shop Floor Control, Journal of Manufacturing
Systems, 21(5), 2002.
Simulation-based Planning Simulation-based Control
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Simulation-based Planning and Control
Y. Son, S. Joshi, R. Wysk, and J. Smith, Simulation Based Shop Floor Control, Journal of Manufacturing
Systems, 21(5), 2002.
Simulation-based Planning
Traditional simulation
(current status)
Reduction
of decision
space (NN)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Real-Time Decision Problems in a Flexible Shop
• Part releasing problem
• (Pull system) Part selection problem
• Machine selection problem
• Part buffering problem
• Robot location problem
• And, other problems
H. Cho, Y. Son, A. Jones, Design and Conceptual Development of Shop Floor Controllers through the
Manipulation of Process Plans, International Journal of Computer Integrated Manufacturing, 19(4), 2006.
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Part Selection Problem in a Flexible Shop
Machine A Machine B Machine C
Cell-
dedicated
buffer
Robot
Part 1
Part 2
Part 4
o1
1
4
3 2
o2
o3
o7
o8
sa ja
To be processed
immediately
Part 3 o4
o5
sa
o6
ja
Part
A B
A
B A
B B
C
H. Cho, Y. Son, A. Jones, Design and Conceptual Development of Shop Floor Controllers through the
Manipulation of Process Plans, International Journal of Computer Integrated Manufacturing, 19(4), 2006.
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Machine Selection Problem in a Flexible Shop
Machine A Machine B Machine C
Cell-
dedicated
buffer
Robot 3
1
4 1 2
Part
Part 1
Part 2
Part 4
o1 o2
o3
o7
o8
sa ja
To be processed
immediately
Part 3 o4
o5
so
o6
jo
A B
C
B A
B B
C
H. Cho, Y. Son, A. Jones, Design and Conceptual Development of Shop Floor Controllers through the
Manipulation of Process Plans, International Journal of Computer Integrated Manufacturing, 19(4), 2006.
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Simulation-based Planning and Control
Y. Son, S. Joshi, R. Wysk, and J. Smith, Simulation Based Shop Floor Control, Journal of Manufacturing
Systems, 21(5), 2002.
Simulation-based Control
• Real-time clock
(parallelism)
• Task generation
• Message exchanges
with FSA
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Simulation-based Planning and Control
Formal model
Automatic Code Generation
for Controllers
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Interactions among controllers
pick put open grasp.....
RT Simulation
(Task Generator)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Interactions among controllers
pick put open grasp.....
RT Simulation
(Task Generator)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Interaction between Planning and Control
H. Cho, Y. Son, A. Jones, Design and Conceptual Development of Shop Floor Controllers through the
Manipulation of Process Plans, International Journal of Computer Integrated Manufacturing, 19(4), 2006.
• Based on same initial conditions (system status)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Hierarchical Process Plans (Cho et al, 2006)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Extension to Top Floor
• Analogies between shop floor and top floor in terms of system
components
– Physical entities and tasks
– Coordination model
– Simulation model
– Resource model
J. Venkateswaran and Y. Son, Hybrid System Dynamic – Discrete Event Simulation based
Architecture for Hierarchical Production Planning, International Journal of Production
Research, 43 (20), 2005.
S. Lee, Y. Son, and R. Wysk, Simulation Based Planning and Control: From Shop Floor to Top
Floor, Journal of Manufacturing Systems, 26 (2), 2007.
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
DDDAMS: Top Floor
• Simulation (fast-mode simulator, and task generator) for
a highly complex system
– Adaptive and localized abstraction during the simulation run
– Simulation steers the measurement process (affecting
model fidelity)
Traditional DDDAMSN. Celik, S. Lee, K. Vasudevan, and Y. Son, DDDAS-based Multi-fidelity Simulation Framework for
Supply Chain Systems, IIE Transactions on Operations Engineering, 42(5), 2010.
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Sensory data
Predetermined
fidelity levelAssigned fidelity
level (δt)
Data filtering algorithm
Algorithm 1
Fidelity selection algorithm
Algorithm 2
Algorithm 4
Algorithm 3
Fidelity assigning algorithm
Real System
Machine 2Machine 1 Machine 3 Machine n
. . .
Information request
Dynamic simulation model reconstruction
algorithm
Operations scheduling
(Task generation)
Information update
Filtered data/
Detected abnormality
Available computational
resource
DDDAS
Data flow
Control flow
Assigned fidelity level (δt)
Embedded Algorithms in DDDAMS Simulation
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Real System
RTdddasF12
C1 C2 C3 C4 C5
Communication Unit
F1
F2
Algorithms
Fidelity Info.=[23322]
Shop
CreatedAdvanced
Models (In different Arena files)
RSPr_RRSPh_RRSM_RRSE_RRSD_R
RSPr_SRSPh_SRSM_SRSE_SRSD_S
RTShop1_R
RTD_RRTE_R
RTM_RRTPh_R
RTPr_R
Information Request
Information Update
Simulation Results
Firing Up Advanced Models
RTdddasC1F3
RTdddasC2F3
RTdddasC3F3
RTdddasC4F3
RTdddasC5F3 RTF3Pr_R
RTF3M_R
RTF3Ph_R
RTF3E_R
RTF3D_R
F3 RTdddasF12
Interaction/Communication Instance
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Surveillance via UAVs/UGVs (or Patrol Agent)
Goal: Develop a simulation-based planning and control system for
surveillance and crowd control via collaborative UAVs/UGVs (Border
Patrol Agent)
Motivation: TUS 1- Project (23-mile long area of southern border in Sasabe, AZ)
Problem: A highly complex, uncertain, dynamically changing border environment
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
DDDAMS-based Planning and Control Framework
Khaleghi, A. M., Xu, D., Wang, Z., Li, M., Lobos, A., Liu, J., & Son, Y. (2013). A DDDAMS-based Planning and Control
Framework for Surveillance and Crowd Control via UAVs and UGVs. Expert Systems with Applications, 40, 7168-7183.
20
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson 21
Multi-resolution Data (Cloud; Trees; Rocks)
Challenge: Aggregate Multi-resolution data
Opportunity: UAVs’ global perception and UGVs’ detailed perception
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Crowd Representation
Next waypoints of UGV
Low density region
invisible by UAV
Representation of crowd individuals
Crowd view by UAV
Crowd view by UGV
Coarser visibility cell
View of dense crowd regions
Finer visibility cell
View of individuals
Next waypoints of UAVs
High density region
visible by UAV
High resolution
Low resolution
Hybrid
Control command generation
for UAV/UGV motion planning
8
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Framework (Planning and Control)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
DR(x)
DR(y)
FOV (x)FOV(y)
h
Detection Module Testbed
24
DR
min
G( )= 2 h
min
G( ) tan FOV / 2( ) £ DR
G( ) £ 2 h
max
G( ) tan FOV / 2( ) = DR
max
G( )
DR
( A) = 2h( A) tan FOV / 2( )
GoPro HERO 3- Tarot Gimbal Stabilizer
HD (16:9): 1280x720p @ 120 ~ 25 fps
FOV(x): 64.4 ; FOV(y): 37.2
EDD : UGV’s Effective Detection Depth
: Detection range for UGV
: Detection range for UAV
: Field of view
: Distance
: Altitude
DRG( )
DRA( )
FOV
h( A)
h(G )
ODROID USB-CAM 720P
HD (16:9): 1280x720p @ 30 fps
FOV: 72
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
1. Shi, J., & Tomasi, C. (1994, June). Good features to track. In Computer Vision and Pattern Recognition, 1994. Proceedings CVPR'94., 1994 IEEE Computer
Society Conference on (pp. 593-600). IEEE.
2. Lucas, B. D., & Kanade, T. (1981, August). An iterative image registration technique with an application to stereo vision. In IJCAI (Vol. 81, pp. 674-679).
Crowd Detection Module UAV
: Applying thresholds
to extract targets of our
interest
25
Optical-Flow-Based Motion Detection
Feature
extraction
Feature
tracking
Warp
successive
frames
Track general
movements
Detect the
target
: Good features to track 1
Extract key points
: Pyramidal Lucas-Kanade 2
Find optical flow vector
: Affine transformation
Warp successive frames
: Motion segmentation
Using the motion history
OpenCV
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Individual Detection Module UGV
: Linear SVM
26
HOG1-Based Human Classification
: 3x3 derivative mask
[-1, 0 , 1]
: Weighted voting over cells
6x6 pixel cells
: Grouping cells into blocks
3x3 cell blocks
: L2-norm
Gradient
computation
Orientation
binning
HOG over des.
blocks
Block
Normalization
Classify the
target
1. Dalal, N., & Triggs, B. (2005, June). Histograms of oriented gradients for human detection. In Computer Vision and
Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on (Vol. 1, pp. 886-893). IEEE.
OpenCV
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Different Crowd Detection Scenarios
27
1. Crowd Joining 2. Crowd Splitting
3. Out of Detection Range 4. Random Movements
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Framework of Proposed Methodology
28
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Crowd Tracking Module
UGV Negligible observation
error
With
observation error
Low
resolution
UGV autoregressive model
-> forecasting
UAV aggregation model ->
prediction
UGV state space model ->
filtering
UAV aggregation model ->
prediction
High
Resolution
UGV autoregressive model
-> forecasting
UAV state space model ->
filtering
UGV state space model ->
filtering
UAV state space model ->
filtering
• Crowd Dynamics Identification
• Crowd Motion Prediction
UAV
7
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson 30
Crowd representation by UAV
Crowd Center Individual Representation Grid
Representation
• Information
loss
• Computationally
intensive
• Require high resolution
• Comprehensive
• Flexible
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson 31
UAVs’ crowd dynamic model
????
The probability of occupancy depends on neighborhoods:
31
?t=t1+4τ
Neighborhood
Dependency
?Estimate by matching
history neighborhood
pattern
𝑷𝒍,𝒎𝑨 𝒕 + 𝝉Pattern to Cell
Occupancy
Cell
Occupancy
Prediction
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson 32
UGVs’ crowd dynamic model
• High-resolution data, Auto-regressive model
Individual
Location
Prediction
𝑷𝒍,𝒎𝑮 𝒕 + 𝝉
Cell
Occupancy
Prediction
Threshold λ
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson 33
Information aggregation
33
𝑃𝑙,𝑚𝑆 𝑡 + 𝜏 = 𝑤𝑙,𝑚
𝑃𝑙,𝑚𝐴 𝑡 + 𝜏 + (1 − 𝑤𝑙,𝑚) 𝑃𝑙,𝑚
𝐺 𝑡 + 𝜏
Prediction based on
UAVs’ dataPrediction based on
UGVs’ data
Aggregated Prediction
𝑷𝟒,𝟒𝑨 𝒕 + 𝝉 =0.33 𝑷𝟒,𝟒
𝑮 𝒕 + 𝝉 =0.95
𝑷𝟒,𝟒𝑺 𝒕 + 𝝉 =0.76 (In this example 𝑤3,3 = 0.3)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson 34
Case Study (Testbed)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson 35
Case Study
35
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Framework of Proposed Methodology
36
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Motion Planning Module• Input:
- Destination: predicted crowd location at time t+Δt
- Map (grid): constructed by discretizing the environment based on terrain elevation
• Output: optimal trajectory (GPS waypoints) to the destination for UAVs and UGVs
Graph Search Algorithms Method Complexity Features
Dijkstra Exact cost from start point to any vertex n high Not fast enough
Best-First SearchEstimated cost from vertex n to the
destionationLow Not optimal
A* Total cost from start point to destinationt lowGuarantees the shortest
path in a reasonable time
• Implementation: Graph search algorithm (A*) with 8-point connectivity and multi objectives
37
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Control Strategies associated with Motion Planning
• Given selected destination of UAV/UGV, find the path that
optimizes a certain combination of criteria
• The generic normalization formula
• Weighted average of the multiple objectives
(c) minimize the weighted
average of (a) and (b)
38
(a) minimize travel distance (b) minimize elevation
penalty (fuel consumption)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
System Implementation
• Agent-based HIL simulation
• UAVs and UGVs
• Social force model and GIS
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Agent-based Hardware-in-the-loop SimulationAgent-based Simulation
Repast Simphony
with 3D GIS
Wi-Fi / XBee PRO 900HP; APM one
Assembled UAV
(APM:Copter / Arducopter)
Assembled UGV
(APM:Rover / Ardurover)
Sensory Data
(e.g. GPS)
Control Commands
(MAVLink Messages)
Hardware Interface:
MAVproxy
Khaleghi, A. M., Xu, D., Lobos, A., Minaeian, S., Son, Y. -J., & Liu, J. (2013). Agent- based hardware-in-the-loop simulation for modeling UAV/UGV
surveillance and crowd control system. In Proceedings of the winter simulation conference 2013, Washington, DC, USA.
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Assembled UAV (Arducopter), AR.Drone, X8+ UAV, and UGV
Navigate GPS waypoints autonomously using APM autopilot set (Arduino-based Autopilot- APM 2.5)
Motion Processing Unit (MPU-6000)
3-Axis Gyro
3-Axis Accelerometer
Microcontroller(ATMEGA2560)
Low-power Atmel 8-bit AVR RISC-based
256KB ISP flash memory
8KB SRAM
4KB EPROM
Throughput: 16 MIPS at 16MHz
Barometric pressure sensor
(SM5611)
GPS update rate: 5hz (5X per second)
Using GPS unit, the UAV has an outdoor navigation accuracy of about +/- 5 meters
Global Positioning System (GPS)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Agent-based Simulation Features
“The Great Circle Distance” (the shortest distance over the earth’s surface) between
any two points using haversine distant equation (Sinnott, 1984): d = Rc
R : Earth's radius
c = 2arctan(hav(c)
1- hav(c))
hav(c) = sin2(f
2-f
1
2)+cosf
1cosf
2sin2(
l2- l
1
2)
(f,l,h) (x, y, z)
42
• Environment enhancement to generate a terrain elevation grid using GIS
• Implementation
• Toolkit: Repast Simphony (Open source, Java, NASA World Wind SDK)
• Mapping GIS Environment to 3D Cartesian (convert to
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Social force model for crowd motion
43
• Calculation of velocity for each agent given 1) desired
destination and 2) force (Helbing et al., 2000)
0 0
( )
( ) ( ) ( )i i i ii i ij iW
j i Wi
d v t t tm m
dt
v e vf f f
Desired vs. current Interaction
with other
agent
Force
against wall
Xi, H., and Y. Son (2010), An integrated pedestrian behavior model
based on extended decision field theory and social force model, In
Human-in-the-loop simulation: Methods and practice. eds. L.
Rothrock, and S. Narayanan: Springer (accepted)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Current Extension
• New challenges
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
3-Level Surveillance Framework
High altitude
level
(HAL)
Low altitude
level (LAL)
Surface level (SL)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
High altitude level (HAL)
Electro-Optical/Infrared
(EO/IR)
Synthetic Aperture Radar
(SAR)
Low altitude level (LAL)
Remote Sensing
Surface level (SL)
Mobile Sensors
Fixed Sensors
3-Level Measurement System in Border Surveillance
Surveillance Camera
3-Levels Types Sensors Measurement Data
SAR
Image
EO/IR
Image
Lidar
Image
Thermal
Images
Magnetic
Data
Spectral
Image
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
A generic modeling framework of sensors for the surveillance application
Adjust the Detection
range;
Set the threshold
Receive signals /
messages from the
controller
Change location
parameters of sensors
in order for chasing foe
targets
Target appears in the
system
Surveillance Behavior
Sub-Model
Availability Sub-
Model
Signal Processing
Sub-Model
Agent Model of Sensors (Generic)
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Bratman, 1987
Rao and Georgeff, 1998
Zhao and Son, 2008
Extended Belief-Desire-Intention Framework
Lee, S., Y.-J. Son, and J. Jin (2010), Integrated human decision making and planning model under extended belief-desire-intention framework,
ACM Transactions on Modeling and Computer Simulation, 20(4), 23(1)~23(24).
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Time
Location
Route Choice En-Route Planning
Fastest Way
Rugged Road
Selection at
Departure
Hiding/Stopping
Sudden Direction
Change
Decision 1 Decision 2 Decision 3
Model of Drug Traffickers based on BDI framework
Behavioral Models of Drug Traffickers
Behavior Models of
Border Patrols
Environment Conditions:
weather, terrains3-Level Sensor Networks
Developed behavior models of drug traffickers and ground patrol agents
together with environmental conditions will provide richer scenarios
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
Summary and Ongoing Works
• Dynamic Data Driven Simulation based
Planning and Control
– Flexible shop floor control
– Surveillance via UAVs/UGVs (or Patrol Agent)
• Ongoing/future Works
– Different control architectures (more centralized;
more decentralized)
– Extension to 3 level hierarchy (UAVs, UGVs,
Aerostat)
– Team formations (rainy days; more mountainous
terrains)
50
Computer Integrated Manufacturing & Simulation Lab
Department of Systems and Industrial Engineering, The University of Arizona, Tucson
QUESTIONS
• Young-Jun Son; [email protected]
• http://www.sie.arizona.edu/faculty/son/index.html
• 1-520-626-9530