dddams-based planning and control - · pdf fileprogram manager: dr. frederica darema. computer...

DDDAMS-based Planning and

Control

Young-Jun Son (son@sie.arizona.edu)

Systems and Industrial Engineering

University of Arizona

Co-PIs: Jian Liu, Jyh-Ming Lien (George Mason)

Students: S. Minaeian, S. Lee, Y. Yuan, A. Khaleghi, D. Xu, Z. Wang, M. Li, N. Celik, K. Vasudevan

Sponsors:

AFOSR: FA9550-12-1-0238 (DDDAS); FA9550-17-

1-0075 (DDDAS); NSF-CNS: 0540212

Program Manager: Dr. Frederica Darema

Computer Integrated Manufacturing & Simulation Lab

Department of Systems and Industrial Engineering, The University of Arizona, Tucson

Theme and Topic

• (Workshop) Towards an Ecosystem of Simulation Models and Data

• (Topic) Online, data-driven simulations– Internet/Integration of things (data, model, embedded

algorithms, and hardware/human)

Simulation-Based Planning and Control

• Simulation-based planning and real-time control (shop floor control) (since late 1990’s)

• Extension to top level supply chain activities via DDDAMS

– Via an NSF project (2005~2007)

• UAV/UGV control via DDDAMS

– Via two AFOSR projects (2012~current)

Simulation-based Planning and Control

Y. Son, S. Joshi, R. Wysk, and J. Smith, Simulation Based Shop Floor Control, Journal of Manufacturing

Systems, 21(5), 2002.

Simulation-based Planning Simulation-based Control

Systems, 21(5), 2002.

Simulation-based Planning

Traditional simulation

(current status)

Reduction

of decision

space (NN)

Real-Time Decision Problems in a Flexible Shop

• Part releasing problem

• (Pull system) Part selection problem

• Machine selection problem

• Part buffering problem

• Robot location problem

• And, other problems

H. Cho, Y. Son, A. Jones, Design and Conceptual Development of Shop Floor Controllers through the

Manipulation of Process Plans, International Journal of Computer Integrated Manufacturing, 19(4), 2006.

Part Selection Problem in a Flexible Shop

Machine A Machine B Machine C

dedicated

buffer

Part 1

Part 2

Part 4

To be processed

immediately

Part 3 o4

Machine Selection Problem in a Flexible Shop

Machine A Machine B Machine C

dedicated

buffer

Robot 3

Part 1

Part 2

Part 4

To be processed

immediately

Part 3 o4

Systems, 21(5), 2002.

Simulation-based Control

• Real-time clock

(parallelism)

• Task generation

• Message exchanges

with FSA

Formal model

Automatic Code Generation

for Controllers

Interactions among controllers

pick put open grasp.....

RT Simulation

(Task Generator)

Interactions among controllers

pick put open grasp.....

RT Simulation

(Task Generator)

Interaction between Planning and Control

• Based on same initial conditions (system status)

Hierarchical Process Plans (Cho et al, 2006)

Extension to Top Floor

• Analogies between shop floor and top floor in terms of system

components

– Physical entities and tasks

– Coordination model

– Simulation model

– Resource model

J. Venkateswaran and Y. Son, Hybrid System Dynamic – Discrete Event Simulation based

Architecture for Hierarchical Production Planning, International Journal of Production

Research, 43 (20), 2005.

S. Lee, Y. Son, and R. Wysk, Simulation Based Planning and Control: From Shop Floor to Top

Floor, Journal of Manufacturing Systems, 26 (2), 2007.

DDDAMS: Top Floor

• Simulation (fast-mode simulator, and task generator) for

a highly complex system

– Adaptive and localized abstraction during the simulation run

– Simulation steers the measurement process (affecting

model fidelity)

Traditional DDDAMSN. Celik, S. Lee, K. Vasudevan, and Y. Son, DDDAS-based Multi-fidelity Simulation Framework for

Supply Chain Systems, IIE Transactions on Operations Engineering, 42(5), 2010.

Sensory data

Predetermined

fidelity levelAssigned fidelity

level (δt)

Data filtering algorithm

Algorithm 1

Fidelity selection algorithm

Algorithm 2

Algorithm 4

Algorithm 3

Fidelity assigning algorithm

Real System

Machine 2Machine 1 Machine 3 Machine n

Information request

Dynamic simulation model reconstruction

algorithm

Operations scheduling

(Task generation)

Information update

Filtered data/

Detected abnormality

Available computational

resource

Data flow

Control flow

Assigned fidelity level (δt)

Embedded Algorithms in DDDAMS Simulation

Real System

RTdddasF12

C1 C2 C3 C4 C5

Communication Unit

Algorithms

Fidelity Info.=[23322]

CreatedAdvanced

Models (In different Arena files)

RSPr_RRSPh_RRSM_RRSE_RRSD_R

RSPr_SRSPh_SRSM_SRSE_SRSD_S

RTShop1_R

RTD_RRTE_R

RTM_RRTPh_R

RTPr_R

Information Request

Information Update

Simulation Results

Firing Up Advanced Models

RTdddasC1F3

RTdddasC2F3

RTdddasC3F3

RTdddasC4F3

RTdddasC5F3 RTF3Pr_R

RTF3M_R

RTF3Ph_R

RTF3E_R

RTF3D_R

F3 RTdddasF12

Interaction/Communication Instance

Surveillance via UAVs/UGVs (or Patrol Agent)

Goal: Develop a simulation-based planning and control system for

surveillance and crowd control via collaborative UAVs/UGVs (Border

Patrol Agent)

Motivation: TUS 1- Project (23-mile long area of southern border in Sasabe, AZ)

Problem: A highly complex, uncertain, dynamically changing border environment

DDDAMS-based Planning and Control Framework

Khaleghi, A. M., Xu, D., Wang, Z., Li, M., Lobos, A., Liu, J., & Son, Y. (2013). A DDDAMS-based Planning and Control

Framework for Surveillance and Crowd Control via UAVs and UGVs. Expert Systems with Applications, 40, 7168-7183.

Department of Systems and Industrial Engineering, The University of Arizona, Tucson 21

Multi-resolution Data (Cloud; Trees; Rocks)

Challenge: Aggregate Multi-resolution data

Opportunity: UAVs’ global perception and UGVs’ detailed perception

Crowd Representation

Next waypoints of UGV

Low density region

invisible by UAV

Representation of crowd individuals

Crowd view by UAV

Crowd view by UGV

Coarser visibility cell

View of dense crowd regions

Finer visibility cell

View of individuals

Next waypoints of UAVs

High density region

visible by UAV

High resolution

Low resolution

Hybrid

Control command generation

for UAV/UGV motion planning

Framework (Planning and Control)

FOV (x)FOV(y)

Detection Module Testbed

G( )= 2 h

G( ) tan FOV / 2( ) £ DR

G( ) £ 2 h

G( ) tan FOV / 2( ) = DR

( A) = 2h( A) tan FOV / 2( )

GoPro HERO 3- Tarot Gimbal Stabilizer

HD (16:9): 1280x720p @ 120 ~ 25 fps

FOV(x): 64.4 ; FOV(y): 37.2

EDD : UGV’s Effective Detection Depth

: Detection range for UGV

: Detection range for UAV

: Field of view

: Distance

: Altitude

DRG( )

DRA( )

ODROID USB-CAM 720P

HD (16:9): 1280x720p @ 30 fps

FOV: 72

1. Shi, J., & Tomasi, C. (1994, June). Good features to track. In Computer Vision and Pattern Recognition, 1994. Proceedings CVPR'94., 1994 IEEE Computer

Society Conference on (pp. 593-600). IEEE.

2. Lucas, B. D., & Kanade, T. (1981, August). An iterative image registration technique with an application to stereo vision. In IJCAI (Vol. 81, pp. 674-679).

Crowd Detection Module UAV

: Applying thresholds

to extract targets of our

interest

Optical-Flow-Based Motion Detection

Feature

extraction

Feature

tracking

successive

frames

Track general

movements

Detect the

target

: Good features to track 1

Extract key points

: Pyramidal Lucas-Kanade 2

Find optical flow vector

: Affine transformation

Warp successive frames

: Motion segmentation

Using the motion history

OpenCV

Individual Detection Module UGV

: Linear SVM

HOG1-Based Human Classification

: 3x3 derivative mask

[-1, 0 , 1]

: Weighted voting over cells

6x6 pixel cells

: Grouping cells into blocks

3x3 cell blocks

: L2-norm

Gradient

computation

Orientation

binning

HOG over des.

blocks

Normalization

Classify the

target

1. Dalal, N., & Triggs, B. (2005, June). Histograms of oriented gradients for human detection. In Computer Vision and

Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on (Vol. 1, pp. 886-893). IEEE.

OpenCV

Different Crowd Detection Scenarios

1. Crowd Joining 2. Crowd Splitting

3. Out of Detection Range 4. Random Movements

Framework of Proposed Methodology

Crowd Tracking Module

UGV Negligible observation

observation error

resolution

UGV autoregressive model

-> forecasting

UAV aggregation model ->

prediction

UGV state space model ->

filtering

UAV aggregation model ->

prediction

Resolution

UGV autoregressive model

-> forecasting

UAV state space model ->

filtering

UGV state space model ->

filtering

UAV state space model ->

filtering

• Crowd Dynamics Identification

• Crowd Motion Prediction

Crowd representation by UAV

Crowd Center Individual Representation Grid

Representation

• Information

• Computationally

intensive

• Require high resolution

• Comprehensive

• Flexible

UAVs’ crowd dynamic model

？？？？

The probability of occupancy depends on neighborhoods:

?t=t1+4τ

Neighborhood

Dependency

?Estimate by matching

history neighborhood

pattern

𝑷𝒍,𝒎𝑨 𝒕 + 𝝉Pattern to Cell

Occupancy

Prediction

UGVs’ crowd dynamic model

• High-resolution data, Auto-regressive model

Individual

Location

Prediction

𝑷𝒍,𝒎𝑮 𝒕 + 𝝉

Occupancy

Prediction

Threshold λ

Information aggregation

𝑃𝑙,𝑚𝑆 𝑡 + 𝜏 = 𝑤𝑙,𝑚

𝑃𝑙,𝑚𝐴 𝑡 + 𝜏 + (1 − 𝑤𝑙,𝑚) 𝑃𝑙,𝑚

𝐺 𝑡 + 𝜏

Prediction based on

UAVs’ dataPrediction based on

UGVs’ data

Aggregated Prediction

𝑷𝟒,𝟒𝑨 𝒕 + 𝝉 =0.33 𝑷𝟒,𝟒

𝑮 𝒕 + 𝝉 =0.95

𝑷𝟒,𝟒𝑺 𝒕 + 𝝉 =0.76 (In this example 𝑤3,3 = 0.3)

Case Study (Testbed)

Case Study

Framework of Proposed Methodology

Motion Planning Module• Input:

- Destination: predicted crowd location at time t+Δt

- Map (grid): constructed by discretizing the environment based on terrain elevation

• Output: optimal trajectory (GPS waypoints) to the destination for UAVs and UGVs

Graph Search Algorithms Method Complexity Features

Dijkstra Exact cost from start point to any vertex n high Not fast enough

Best-First SearchEstimated cost from vertex n to the

destionationLow Not optimal

A* Total cost from start point to destinationt lowGuarantees the shortest

path in a reasonable time

• Implementation: Graph search algorithm (A*) with 8-point connectivity and multi objectives

Control Strategies associated with Motion Planning

• Given selected destination of UAV/UGV, find the path that

optimizes a certain combination of criteria

• The generic normalization formula

• Weighted average of the multiple objectives

(c) minimize the weighted

average of (a) and (b)

(a) minimize travel distance (b) minimize elevation

penalty (fuel consumption)

System Implementation

• Agent-based HIL simulation

• UAVs and UGVs

• Social force model and GIS

Agent-based Hardware-in-the-loop SimulationAgent-based Simulation

Repast Simphony

with 3D GIS

Wi-Fi / XBee PRO 900HP; APM one

Assembled UAV

(APM:Copter / Arducopter)

Assembled UGV

(APM:Rover / Ardurover)

Sensory Data

(e.g. GPS)

Control Commands

(MAVLink Messages)

Hardware Interface:

MAVproxy

Khaleghi, A. M., Xu, D., Lobos, A., Minaeian, S., Son, Y. -J., & Liu, J. (2013). Agent- based hardware-in-the-loop simulation for modeling UAV/UGV

surveillance and crowd control system. In Proceedings of the winter simulation conference 2013, Washington, DC, USA.

Assembled UAV (Arducopter), AR.Drone, X8+ UAV, and UGV

Navigate GPS waypoints autonomously using APM autopilot set (Arduino-based Autopilot- APM 2.5)

Motion Processing Unit (MPU-6000)

3-Axis Gyro

3-Axis Accelerometer

Microcontroller(ATMEGA2560)

Low-power Atmel 8-bit AVR RISC-based

256KB ISP flash memory

8KB SRAM

4KB EPROM

Throughput: 16 MIPS at 16MHz

Barometric pressure sensor

(SM5611)

GPS update rate: 5hz (5X per second)

Using GPS unit, the UAV has an outdoor navigation accuracy of about +/- 5 meters

Global Positioning System (GPS)

Agent-based Simulation Features

“The Great Circle Distance” (the shortest distance over the earth’s surface) between

any two points using haversine distant equation (Sinnott, 1984): d = Rc

R : Earth's radius

c = 2arctan(hav(c)

1- hav(c))

hav(c) = sin2(f

2)+cosf

2sin2(

(f,l,h) (x, y, z)

• Environment enhancement to generate a terrain elevation grid using GIS

• Implementation

• Toolkit: Repast Simphony (Open source, Java, NASA World Wind SDK)

• Mapping GIS Environment to 3D Cartesian (convert to

Social force model for crowd motion

• Calculation of velocity for each agent given 1) desired

destination and 2) force (Helbing et al., 2000)

( ) ( ) ( )i i i ii i ij iW

j i Wi

d v t t tm m

v e vf f f

Desired vs. current Interaction

with other

against wall

Xi, H., and Y. Son (2010), An integrated pedestrian behavior model

based on extended decision field theory and social force model, In

Human-in-the-loop simulation: Methods and practice. eds. L.

Rothrock, and S. Narayanan: Springer (accepted)

Current Extension

• New challenges

3-Level Surveillance Framework

High altitude

Low altitude

level (LAL)

Surface level (SL)

High altitude level (HAL)

Electro-Optical/Infrared

(EO/IR)

Synthetic Aperture Radar

Low altitude level (LAL)

Remote Sensing

Surface level (SL)

Mobile Sensors

Fixed Sensors

3-Level Measurement System in Border Surveillance

Surveillance Camera

3-Levels Types Sensors Measurement Data

Thermal

Images

Magnetic

Spectral

A generic modeling framework of sensors for the surveillance application

Adjust the Detection

range;

Set the threshold

Receive signals /

messages from the

controller

Change location

parameters of sensors

in order for chasing foe

targets

Target appears in the

system

Surveillance Behavior

Sub-Model

Availability Sub-

Signal Processing

Sub-Model

Agent Model of Sensors (Generic)

Bratman, 1987

Rao and Georgeff, 1998

Zhao and Son, 2008

Extended Belief-Desire-Intention Framework

Lee, S., Y.-J. Son, and J. Jin (2010), Integrated human decision making and planning model under extended belief-desire-intention framework,

ACM Transactions on Modeling and Computer Simulation, 20(4), 23(1)~23(24).

Location

Route Choice En-Route Planning

Fastest Way

Rugged Road

Selection at

Departure

Hiding/Stopping

Sudden Direction

Change

Decision 1 Decision 2 Decision 3

Model of Drug Traffickers based on BDI framework

Behavioral Models of Drug Traffickers

Behavior Models of

Border Patrols

Environment Conditions:

weather, terrains3-Level Sensor Networks

Developed behavior models of drug traffickers and ground patrol agents

together with environmental conditions will provide richer scenarios

Summary and Ongoing Works

• Dynamic Data Driven Simulation based

Planning and Control

– Flexible shop floor control

– Surveillance via UAVs/UGVs (or Patrol Agent)

• Ongoing/future Works

– Different control architectures (more centralized;

more decentralized)

– Extension to 3 level hierarchy (UAVs, UGVs,

Aerostat)

– Team formations (rainy days; more mountainous

terrains)

QUESTIONS

• Young-Jun Son; son@sie.arizona.edu

• http://www.sie.arizona.edu/faculty/son/index.html

• 1-520-626-9530

dddams-based planning and control - · pdf fileprogram manager: dr. frederica darema. computer...

Documents

reflections on production grids enterprise and research...

a report of the national science...

48 frederica road, winton, bournemouth bh9 2na

pca 350: frederica de laguna photograph...

darema - dynamic data driven applications systems (dddas) -...

frederica academy media center facility plan

no slide title · pdf filedr. frederica darema . 4 afosr...

computer systems research program in the computer and...

1 dr. frederica darema senior science and technology advisor...

itqm 2016 finalized program 2017 program.pdf · frederica...

dilemma - life to the max (frederica)

task force co-chairs from the acci -...

steering complex systems using a dynamic data-driven...

1 dr. frederica darema cise/nsf performance engineering...

morning message - princess frederica home

dr. frederica darema air force office of scientific research

program officers: frederica darema helen gill brett fleisch...

new frontiers through computer and information...

1 dr. frederica darema senior science and technology advisor...

frederica felton sussex county delaware