01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Interactive Simulation and Visualization in City and Regional Planning Uwe Wössner, HLRS

SEAIP 2011

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

THE HLRS SYSTEMS

SEAIP 2011

Fileserver

Stuttgart

NEC SX-8 (576/~120)

DALCO Viz-Cluster (8*4+32*2)

NEC Nehalem (1400x4)

Cray XT5+XD1 (224*4+48)

SUN Fire 2900 (48x2)

Ulm

NEC Asama (64)

CAVE

IBM x3755

(144x2)

IBM (2800x4)

IBM HSM > 2 PB

IBM Cell

NEC SX-9 (156)

Hermit Cray XE6(7000x16)

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

PHASE 1 STEP 1: OVERVIEW

SEAIP 2011

• Configuration:

– Peak Performace ~ 1PF

– More than 3500 nodes

– Each Node will have 2 sockets

• AMD Interlagos @ 2.3GHz 16 Cores each leading to >100.000 cores

– Nodes with 32GB and 64GB memory reflecting different user needs

– 2.7PB storage capacity @ ~ 150GB/s IO bandwidth

– External Access Nodes, Pre- & Postprocessing Nodes, Remote Visualization Nodes

– ~2MW maximal power consumption

• Essential part of the contract is an intensive support by on-site staff and a collaboration agreement between HLRS and CRAY

• Support for ISV Codes depending on the app under CLE („native“) or CCM

• Part of the acceptance tests are validation of sustained application performance spanning across the full system

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

STORAGE

SEAIP 2011

• Lustre based solution for the fast disk space

– 2.7 PB Lustre Workspace capacity

– Realised with 16 DDN SFA10k controllers

– Integrated into the overall HLRS environments

• HLRS wide Home Space with 60TB capacity

• Local storage capacity with 20TB for Pre- and Postprocessing servers

• Integrated with the existing HPSS system using a data mover concept 3840TB Total Raw Capacity

LNET 1

LNET 68 ……………………….

MDS 2 IB Switch Fabric

7x2+2RAID10 600GB FC

OSS 64 ……………….

MDS 1

OSS 1

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

PRE- AND POSTPROCESSING

SEAIP 2011

• Pre- and Postprocessing servers support users in their workflow with large memory and full Linux environment

• Phase 1 Step 1 will have several Pre-/Postprocessing servers with up to 1TB of main memory

• The following step will add more servers with up to 2TB main memory

• Resources will be also controlled by the scheduling system

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

COVISE COLLABORATIVE VISUALIZATION AND SIMULATION ENVIRONMENT

SEAIP 2011

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

INTERACTIVE SIMULATION IN COVISE

SEAIP 2011

Grid generation

Domain decomposition Simulation

Post processing

Rendering

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

VIRTUAL TURBINE TEST-BED

SEAIP 2011

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

UPDATE ON SIMULATION COUPLING

SEAIP 2011

Code Type Coupling mode

StarCD CFD SimLib, Fortran user subroutine, Loose

CFX CFD SimLib, Fortran user subroutine, Loose

Fluent CFD SimLib, C user subroutine, Loose

OpenFoam CFD SimLib, C++ user subroutine, Loose

Fidap CFD Script, Black-Box, Loose

Fenfloss CFD SimLib, C Plugin, Loose

Diablo Diffusion in metal alloys Script, Black-Box, Loose

Trans3D Laser ablation COVISE Modul C++, Tight

Uranus CFD Socket interface, coupling module, C++, Loose

SunFace Radiation COM/DCOM, coupling module, C++, Hybrid

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

AUGMENTED REALITY

SEAIP 2011

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

FREIGHTLINER CASCADIA (SC11)

SEAIP 2011

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

MODELL WIND TUNNEL AT THE FKFS

SEAIP 2011

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

WIND TUNNEL AT FKFS

SEAIP 2011

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

COLLABORATIVE WIND TUNNEL

SEAIP 2011

Interactive Simulation in Architecture

SEAIP 2011

Stadtraumlabor TU-Wien

SEAIP 2011

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

INTERACTIVE SIMULATION IN ARCHITECTURE

SEAIP 2011

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

ACCESS TO ARCGIS AND WMS SERVERS

SEAIP 2011

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

WATER POWER HISTORY AND POTENTIAL FOR EXTENSION

SEAIP 2011

Upper Stage

Height 324m

Reservoir volume 1,8 Mio m³

Turbine power 200MW Lower Stage

Height 364m

Reservoir volume 575.000 m³

Turbine power 70MW

65,3MW

22MW

2,5MW

0,5MW 14,4 Mio m³

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

SUSTAINABILITY

SEAIP 2011

Economic

Social Environmental

Sustainability

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

SUSTAINABILITY

SEAIP 2011

Tourism

Wildlife preservation

Education

Barrier free Rivers

Forestry

Biodiversity

Electricity

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

PARALLEL POST PROCESSING

SEAIP 2011

MPI

MPI

MPI Renderer

Module 1 Module 1

Module 2

Datamanager Shared Memory

GPU

Renderer

Module 1 Module 1

Module 2

Datamanager Shared Memory

GPU

Renderer

Controller

MPI Transport Layer TCP Transport Layer

Start

MPI_Spawn

• Parallel Rendering

• Remote Rendering

• VR / Desktop

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

GP-GPU POST PROCESSING

SEAIP 2011

Visualization Cluster • Multiple Nodes

• Multiple CPUs / Node

• highly parallel GPUs

Data Processing on

GPUs • Preparation

• Processing

• Rendering / Readback

Datamanager stores

(partial) data objects • Host memory

• Device memory

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

CUDA ISO/CUTTINGSURFACE COMPUTATION

SEAIP 2011

• Classify Elements / number of vertices (TLT)

– Tetrahedron, Hexahedron, Prism

• Exclusive Prefix

Sum (scan)

• Reduce (scatter)

• Interpolate /

• Generate Triangles

3 0 6 0 0 0 0 12 0 0 9 0 1 0 1 0 0 0 0 1 0 0 1 0

0 3 3 9 9 9 9 9 21 21 21 30 0 1 1 2 2 2 2 2 3 3 3 4

21 21 21 30 3 3 3 4

0 3 9 21 0 1 2 3

Isosurface extraction Initialization reduce

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

PARALLEL SURFACE EXTRACTION (GPU)

SEAIP 2011

MPI

MPI

MPI Renderer

Module 1 Module 1

Module 2

Datamanager

Shared Memory

GPU

Renderer

Module 1 Module 1

Module 2

Datamanager

Shared Memory

GPU

MPI Transport Layer

Parallelization of iso-/cutting surface extraction

for interactive post-processing on unstructured

grids

• NVIDIA Quadro 6000 (Fermi):

>5x faster than 16 Xeon E5472

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

INTERACTIVE TEXTURE BASED FLOW VISUALIZATION

SEAIP 2011

Line Integral Convolution • Convolute a noise texture alongside a flow field

• Vector field tracing (Euler/Runge-Kutta)

• Pixels along field lines show strong correlation

• Pixels orthogonal to the field lines are uncorrelated

-> Qualitative view of a slice of the flow field

Interactive speeds through

• Flow field as per-vertex attribute in fragment

shader (obtained from CUDA cutting-surface

kernel)

• Evaluating the flow-field once per fragment

• Convoluting the noise texture along interpolated

vector

High gradient in the flow field -> wrong vectors

(esp. far away from fragment position)

Noise Flow field on

extracted

surface

LIC result

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

INTERACTIVE LIC COMPARISON

SEAIP 2011

64 vector field evaluations

1 vector field evaluation

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

PARALLEL PARTICLE TRACING

SEAIP 2011

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

INITIAL CELL LOCATION / NEIGHBOR SEARCH

SEAIP 2011

• KD-Tree

• Walk to neighboring element

• On-the-fly decomposition to tetrahedra

• Interpolation in tetrahedra

• Runge-Kutta(4)

a

b

(0,1)

(1,0) (0,0)

1-a-b=0

a

b

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

MASSIVE PARTICLE TRACING

SEAIP 2011

Simulation of erosion in a turbine runner

• In postprocessing on a hybrid many-core CPU/GPU cluster

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

MASSIVE PARTICLES

SEAIP 2011

gmmF

UUUUACF

m

FFFF

dt

dU

FPB

PFPFFDD

P

PVMBDP

F

-=

--=

=

)(

)(||2

1

Assumptions: • Spherical, monomorphic particles

• Particles do not influence fluid and walls

• No interaction between particles

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

TABAKOFF MODEL OF EROSION

SEAIP 2011

0

0

2

4

2

3

2

0

122

222

1

2 if 0.0

2 if 1.0

)sin()(

sin1

)]2

sin(1[)(

)()1(cos)()(

{

=

=

-=

=

-=

k

V

VVf

V

VR

kkf

VfRV

VfE

P

PN

P

T

PNT

P

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

EROSION IN TURBINE RUNNERS

SEAIP 2011

Tracing of massive sand particles

CFD Simulation: Ansys CFX

15.215.488 hexahedral elements

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

GP-GPU POST-PROCESSING

SEAIP 2011

• Post-processing of numerical simulation data at interactive speeds on the visualization cluster using GPGPU

• immediate rendering of VBOs

• asynchronous / overlapping streaming memcpy of transient data

• Interactive qualitative visualization of vector data (LIC)

• Integrates with parallel rendering (sort last)

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

:: SEAIP 2011

VALIDATE Virtual Automotive Lab for Integrated Digital Automation Technologies

Driving Simulation at University of Stuttgart

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

PORSCHE 997 TURBO (BEFORE)

SEAIP 2011

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

VALIDATE

SEAIP 2011

•3 degrees of freedom •3 linear motors •a-max 2g •V-max 1m/s •Force feedback steering wheel •Active gas pedal (Conti) •Active brake pedal

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

COMPONENTS OF THE DRIVING SIMULATOR

SEAIP 2011

•Road surface •Street logic •Third party vehicle simulation

•Landscape •Buildings •Traffic signs •Side rails •Trees

•Hardware •Restbus Simulation •Vehicle dynamics

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

THIRD PARTY VEHICLE SIMULATION

SEAIP 2011

•Realistic behavior •Respecting traffic rules •Car to car / car to infrastructure comm. •Pedestrians

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

APPLICATIONS

SEAIP 2011

Lane Change Test

•Standard tests, e.g. lane change test •Evaluation of driver assistance systems •Test of new HMI concepts Entertainment, GPS, Telephone, air conditioning •Driver reaction in critical situations •Visibility studies •Evaluation of rear view mirrors •Evaluation head lights

01.12.2011 :: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

THE HLRS VIS-TEAM

SEAIP 2011

• Blasius Czink – (SCALB)

• Andreas Gottlieb – Automatic optimization (Daimler CFD)

• Andreas Kopecki – Parallel Rendering;Collaborative Working (VisPME/VISIONAIR)

• Florian Niebling – Parallel Visualization; Grid;Infiniband; (VisPME)

• Jutta Sauer – Video streaming; Video Conferencing; COVISE Dev.; 3D Modeling

• Florian Seybold – Driving Simulator; interactive vehicle dynamics (VALIDATE); GP-GPU SPH (IAA)

• Uwe Zimmat – Video service

• N.N. – Parallel and Remote Visualization (CRESTA)

• N.N. – Interactive Simulation (Cool-Em-All)

• N.N. – Virtual Reality(VISIONAIR)

czink-gottlieb-kopecki-niebling-sauer-seybold-woessner-zimmat@hlrs.de