01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
Interactive Simulation and Visualization in City and Regional Planning Uwe Wössner, HLRS
SEAIP 2011
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
THE HLRS SYSTEMS
SEAIP 2011
Fileserver
Stuttgart
NEC SX-8 (576/~120)
DALCO Viz-Cluster (8*4+32*2)
NEC Nehalem (1400x4)
Cray XT5+XD1 (224*4+48)
SUN Fire 2900 (48x2)
Ulm
NEC Asama (64)
CAVE
IBM x3755
(144x2)
IBM (2800x4)
IBM HSM > 2 PB
IBM Cell
NEC SX-9 (156)
Hermit Cray XE6(7000x16)
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
PHASE 1 STEP 1: OVERVIEW
SEAIP 2011
• Configuration:
– Peak Performace ~ 1PF
– More than 3500 nodes
– Each Node will have 2 sockets
• AMD Interlagos @ 2.3GHz 16 Cores each leading to >100.000 cores
– Nodes with 32GB and 64GB memory reflecting different user needs
– 2.7PB storage capacity @ ~ 150GB/s IO bandwidth
– External Access Nodes, Pre- & Postprocessing Nodes, Remote Visualization Nodes
– ~2MW maximal power consumption
• Essential part of the contract is an intensive support by on-site staff and a collaboration agreement between HLRS and CRAY
• Support for ISV Codes depending on the app under CLE („native“) or CCM
• Part of the acceptance tests are validation of sustained application performance spanning across the full system
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
STORAGE
SEAIP 2011
• Lustre based solution for the fast disk space
– 2.7 PB Lustre Workspace capacity
– Realised with 16 DDN SFA10k controllers
– Integrated into the overall HLRS environments
• HLRS wide Home Space with 60TB capacity
• Local storage capacity with 20TB for Pre- and Postprocessing servers
• Integrated with the existing HPSS system using a data mover concept 3840TB Total Raw Capacity
LNET 1
LNET 68 ……………………….
MDS 2 IB Switch Fabric
7x2+2RAID10 600GB FC
OSS 64 ……………….
MDS 1
OSS 1
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
PRE- AND POSTPROCESSING
SEAIP 2011
• Pre- and Postprocessing servers support users in their workflow with large memory and full Linux environment
• Phase 1 Step 1 will have several Pre-/Postprocessing servers with up to 1TB of main memory
• The following step will add more servers with up to 2TB main memory
• Resources will be also controlled by the scheduling system
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
COVISE COLLABORATIVE VISUALIZATION AND SIMULATION ENVIRONMENT
SEAIP 2011
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
INTERACTIVE SIMULATION IN COVISE
SEAIP 2011
Grid generation
Domain decomposition Simulation
Post processing
Rendering
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
VIRTUAL TURBINE TEST-BED
SEAIP 2011
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
UPDATE ON SIMULATION COUPLING
SEAIP 2011
Code Type Coupling mode
StarCD CFD SimLib, Fortran user subroutine, Loose
CFX CFD SimLib, Fortran user subroutine, Loose
Fluent CFD SimLib, C user subroutine, Loose
OpenFoam CFD SimLib, C++ user subroutine, Loose
Fidap CFD Script, Black-Box, Loose
Fenfloss CFD SimLib, C Plugin, Loose
Diablo Diffusion in metal alloys Script, Black-Box, Loose
Trans3D Laser ablation COVISE Modul C++, Tight
Uranus CFD Socket interface, coupling module, C++, Loose
SunFace Radiation COM/DCOM, coupling module, C++, Hybrid
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
AUGMENTED REALITY
SEAIP 2011
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
FREIGHTLINER CASCADIA (SC11)
SEAIP 2011
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
MODELL WIND TUNNEL AT THE FKFS
SEAIP 2011
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
WIND TUNNEL AT FKFS
SEAIP 2011
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
COLLABORATIVE WIND TUNNEL
SEAIP 2011
Interactive Simulation in Architecture
SEAIP 2011
Stadtraumlabor TU-Wien
SEAIP 2011
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
INTERACTIVE SIMULATION IN ARCHITECTURE
SEAIP 2011
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
ACCESS TO ARCGIS AND WMS SERVERS
SEAIP 2011
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
WATER POWER HISTORY AND POTENTIAL FOR EXTENSION
SEAIP 2011
Upper Stage
Height 324m
Reservoir volume 1,8 Mio m³
Turbine power 200MW Lower Stage
Height 364m
Reservoir volume 575.000 m³
Turbine power 70MW
65,3MW
22MW
2,5MW
0,5MW 14,4 Mio m³
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
SUSTAINABILITY
SEAIP 2011
Economic
Social Environmental
Sustainability
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
SUSTAINABILITY
SEAIP 2011
Tourism
Wildlife preservation
Education
Barrier free Rivers
Forestry
Biodiversity
Electricity
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
PARALLEL POST PROCESSING
SEAIP 2011
MPI
MPI
MPI Renderer
Module 1 Module 1
Module 2
Datamanager Shared Memory
GPU
Renderer
Module 1 Module 1
Module 2
Datamanager Shared Memory
GPU
Renderer
Controller
MPI Transport Layer TCP Transport Layer
Start
MPI_Spawn
• Parallel Rendering
• Remote Rendering
• VR / Desktop
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
GP-GPU POST PROCESSING
SEAIP 2011
Visualization Cluster • Multiple Nodes
• Multiple CPUs / Node
• highly parallel GPUs
Data Processing on
GPUs • Preparation
• Processing
• Rendering / Readback
Datamanager stores
(partial) data objects • Host memory
• Device memory
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
CUDA ISO/CUTTINGSURFACE COMPUTATION
SEAIP 2011
• Classify Elements / number of vertices (TLT)
– Tetrahedron, Hexahedron, Prism
• Exclusive Prefix
Sum (scan)
• Reduce (scatter)
• Interpolate /
• Generate Triangles
3 0 6 0 0 0 0 12 0 0 9 0 1 0 1 0 0 0 0 1 0 0 1 0
0 3 3 9 9 9 9 9 21 21 21 30 0 1 1 2 2 2 2 2 3 3 3 4
21 21 21 30 3 3 3 4
0 3 9 21 0 1 2 3
Isosurface extraction Initialization reduce
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
PARALLEL SURFACE EXTRACTION (GPU)
SEAIP 2011
MPI
MPI
MPI Renderer
Module 1 Module 1
Module 2
Datamanager
Shared Memory
GPU
Renderer
Module 1 Module 1
Module 2
Datamanager
Shared Memory
GPU
MPI Transport Layer
Parallelization of iso-/cutting surface extraction
for interactive post-processing on unstructured
grids
• NVIDIA Quadro 6000 (Fermi):
>5x faster than 16 Xeon E5472
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
INTERACTIVE TEXTURE BASED FLOW VISUALIZATION
SEAIP 2011
Line Integral Convolution • Convolute a noise texture alongside a flow field
• Vector field tracing (Euler/Runge-Kutta)
• Pixels along field lines show strong correlation
• Pixels orthogonal to the field lines are uncorrelated
-> Qualitative view of a slice of the flow field
Interactive speeds through
• Flow field as per-vertex attribute in fragment
shader (obtained from CUDA cutting-surface
kernel)
• Evaluating the flow-field once per fragment
• Convoluting the noise texture along interpolated
vector
High gradient in the flow field -> wrong vectors
(esp. far away from fragment position)
Noise Flow field on
extracted
surface
LIC result
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
INTERACTIVE LIC COMPARISON
SEAIP 2011
64 vector field evaluations
1 vector field evaluation
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
PARALLEL PARTICLE TRACING
SEAIP 2011
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
INITIAL CELL LOCATION / NEIGHBOR SEARCH
SEAIP 2011
• KD-Tree
• Walk to neighboring element
• On-the-fly decomposition to tetrahedra
• Interpolation in tetrahedra
• Runge-Kutta(4)
a
b
(0,1)
(1,0) (0,0)
1-a-b=0
a
b
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
MASSIVE PARTICLE TRACING
SEAIP 2011
Simulation of erosion in a turbine runner
• In postprocessing on a hybrid many-core CPU/GPU cluster
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
MASSIVE PARTICLES
SEAIP 2011
gmmF
UUUUACF
m
FFFF
dt
dU
FPB
PFPFFDD
P
PVMBDP
F
-=
--=
=
)(
)(||2
1
Assumptions: • Spherical, monomorphic particles
• Particles do not influence fluid and walls
• No interaction between particles
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
TABAKOFF MODEL OF EROSION
SEAIP 2011
0
0
2
4
2
3
2
0
122
222
1
2 if 0.0
2 if 1.0
)sin()(
sin1
)]2
sin(1[)(
)()1(cos)()(
{
=
=
-=
=
-=
k
V
VVf
V
VR
kkf
VfRV
VfE
P
PN
P
T
PNT
P
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
EROSION IN TURBINE RUNNERS
SEAIP 2011
Tracing of massive sand particles
CFD Simulation: Ansys CFX
15.215.488 hexahedral elements
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
GP-GPU POST-PROCESSING
SEAIP 2011
• Post-processing of numerical simulation data at interactive speeds on the visualization cluster using GPGPU
• immediate rendering of VBOs
• asynchronous / overlapping streaming memcpy of transient data
• Interactive qualitative visualization of vector data (LIC)
• Integrates with parallel rendering (sort last)
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: SEAIP 2011
VALIDATE Virtual Automotive Lab for Integrated Digital Automation Technologies
Driving Simulation at University of Stuttgart
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
PORSCHE 997 TURBO (BEFORE)
SEAIP 2011
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
VALIDATE
SEAIP 2011
•3 degrees of freedom •3 linear motors •a-max 2g •V-max 1m/s •Force feedback steering wheel •Active gas pedal (Conti) •Active brake pedal
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
COMPONENTS OF THE DRIVING SIMULATOR
SEAIP 2011
•Road surface •Street logic •Third party vehicle simulation
•Landscape •Buildings •Traffic signs •Side rails •Trees
•Hardware •Restbus Simulation •Vehicle dynamics
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
THIRD PARTY VEHICLE SIMULATION
SEAIP 2011
•Realistic behavior •Respecting traffic rules •Car to car / car to infrastructure comm. •Pedestrians
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
APPLICATIONS
SEAIP 2011
Lane Change Test
•Standard tests, e.g. lane change test •Evaluation of driver assistance systems •Test of new HMI concepts Entertainment, GPS, Telephone, air conditioning •Driver reaction in critical situations •Visibility studies •Evaluation of rear view mirrors •Evaluation head lights
01.12.2011 :: ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
::
THE HLRS VIS-TEAM
SEAIP 2011
• Blasius Czink – (SCALB)
• Andreas Gottlieb – Automatic optimization (Daimler CFD)
• Andreas Kopecki – Parallel Rendering;Collaborative Working (VisPME/VISIONAIR)
• Florian Niebling – Parallel Visualization; Grid;Infiniband; (VisPME)
• Jutta Sauer – Video streaming; Video Conferencing; COVISE Dev.; 3D Modeling
• Florian Seybold – Driving Simulator; interactive vehicle dynamics (VALIDATE); GP-GPU SPH (IAA)
• Uwe Zimmat – Video service
• N.N. – Parallel and Remote Visualization (CRESTA)
• N.N. – Interactive Simulation (Cool-Em-All)
• N.N. – Virtual Reality(VISIONAIR)
czink-gottlieb-kopecki-niebling-sauer-seybold-woessner-zimmat@hlrs.de