Distributed

Interactive

Computing

Environment

http://www.arl.hpc.mil/SciVis/dice

CFD Code 1

CSM Code 1

CCM Code 2

CCM Code 1

Preprocessing

Runtime Analysis

Postprocessing

HPC Codes

Platforms

Tools

Challenge : Stovepiped Applications

Additional Challenges

• Code Setup can be Difficult

• Enormous Amounts of Data

• Heterogeneous Environment

• Security

High PerformanceComputing Resource

User’s Workstation

Graphical UserInterface and Runtime Visualization

Distributed Interactive Computing Environment

Common Distributed Data Model and Format

Fully Integrated FMD GUI

Fully Integrated ZnsFlow GUI

Fully Integrated CTH GUI

What is DICE ?

• Heterogeneous DSM *

• Data Organization

• Graphical User Interface

• Visualization

A Distributed Computing Environment Composed of Configurable Mega-Components

User-Ready Computing Environments Customized for Specific Technology Areas

Process Local Address Space

System Shared Memory Disk File

0 10M 30M 50M

Physical Data Storage

Servers

Clients

Network Distributed Global Memory

Put Get

VectorPut VectorGet

Barrier Semaphore

Light Data

• 3D Structured Mesh

• 300 x 200 x 300

• Number of Materials

• Material Names

• Total Residual Mass

Heavy Data

• X, Y, Z Values

• Material Volume Fractions

• Cell Temperatures

Operating System

Distributed Resources

Network Distributed Global Memory ( NDGM )with GSSAPI

Hierarchical Data Format ( HDF5 )Data Format : Number Type, Data Space ...

DICE Object DirectoryObject Oriented Support for Data Model, XML, ...

High PerformanceComputing Code

VisualizationToolkit (vtk)

OpenGL

Mesa

Visual GeneratorsPlots

PlanesSurfaces

Scripting and Graphical User Interface Tools

User Application

File I/OProcess Control

Shared Memory FacilityTCP/IP Sockets

DICE Data Model and Format

Distributed Buffer of Bytes

Structured Data Unstructured Data Groups

Convenience Layer

Network Distributed Global Memory with

GSSAPI

Hierarchical Data Format ( HDF5 )Data Format : Number Type, Data Space ...

DICE Object DirectoryObject Oriented Support for Data Model, XML, ...

FileSystem Serial and Parallel

Global Access to Secondary Storage ( Globus - GASS )

Other ...

Hierarchical Data Format Version 5

• Host Independent Data Format

• Access via API only

• Physical Storage thru Drivers

• Includes Data Compression

• Includes SubSelections

Group

DataSet

Attribute

Group Group

Group

Group

DataSet

DataSet

DataSet

Attribute

Attribute

Attribute

Attribute Not Allowed

eXtensible Markup Language

• Similar to HTML Format

• Human Readable

• Pervasive on the WEB

• Strict Hierarchy with No Inheritance

• Many Parsing Tools Already Exist

< Parent Name=“Top”>

< Child Name=“Left”>

Character Data for Left

< /Child >

<Child Name=“Right” >

Character Data for Right

< /Child >

< /Parent >

Parent

Child

Character Data for Left

Child

Character Data for Right

Document Object Model

In-Memory Data Structures

XML

Parse

Serialize

1

1 1 1

11 1

1

2 2 2 2

2 2

2

2

3

33

3

3

33

3

4

4 4

4

4

4

4

5

55

5

6

6

78

NN

i

j

ki

j

12

3

4

5N

Polyvertex Polyline Triangle Quadrilateral Polygon

Tetrahedron Pyramid Wedge

Hexahedron 2DSMesh3DSMesh

DICE Object Directory Base Topologies

HDF5

NDGM

HPC Code

GASS MPI-I/O . . . .

<Topology Type=“Tetrahedra” NumElements=“3”>

</Topology>

<Geometry>

<Vertices>

<Index Location=“XML” Type=“List”>

1000 1001 21001 21000 ……

</Index>

<Points Type=“XYZ” Location=“HDF” >

NDGM:File1.h5:/Grid1/Geometry/MyPoints

</Points>

</Vertices>

</Geometry>

<Topology Type=“3DSmesh”

Dimensions=“10 20 30”>

</Topology>

<Geometry>

<Points Type=“XYZ” Location=“HDF” >

NDGM:File1.h5:/Grid1/Geometry/MyPoints

<Region Type=“Slab” Location=“XML” >

1 2 10 2 3 20 3 4 30

</Region>

</Points>

</Geometry>

Light Data

100 x 200 x 300

HDF5

File

Domain

Topology

100 Hexahedra

Grid

Geometry XYZPoints.h5

Attribute

NDGM:Pressure.h5

XML

HPC Code

Domain

HDF5

Complexity of Model Defined in Light Data

HPC Code I/O is Natural and Efficient

NDGM GASS

Attribute

NDGM:Temperature.h5

Grid

HDF5

Domain

Topology

100 Hexahedra

Grid

Geometry XYZPoints.h5

Parameter TimeStep = 0 to 99

Attribute

NDGM:&TimeStep;/Pressure.h5

XML

HPC Code

Domain

HDF5 for TimeStep 0

Grid

XML Contains Parameters for Variable Substitution

HDF5 for TimeStep 1

HDF5 for TimeStep 2

Hierarchical Data Format Verision 5 ( HDF5 )

eXtensible Markup Language ( XML )

HPC Codes

Pre-Processing ToolsPost-Processing Tools

Runtime Tools

Distributed Data Hub

Parallel / Distributed Entities

Convenience Layer

DICE Object Directory

DICE Data Reader for EnSight

Data Reader uses DICE Object Directory API

EnSight

DICE Visualization using vtk

Source

Filter Filter

Filter

Visualization “Subnet”

Polygons

HDF5

File

HDF5

NDGM GASSDICE Object

Directory

System Programming LanguageC C++ Fortran77 Fortran90 etc.

SWIG Compiled Object

Tcl Wrapper

Perl Wrapper

Python Wrapper

Java Wrapper

New Scripting Commandvia

Dynamically Loadable Library}

Adding Performance Friendly Functionality

DiceFloat64Array MyData, NewData;

DiceInt64Array MyIndex;

MyData.SetNumberOfElements( 10000 );

MyData.Generate(0.0, .9999 );

MyIndex.SetNumberOfElements( 100 );

MyIndex.Generate( 0, 99 );

MyIndex *= 2;

NewData = MyData[ MyIndex ];

DiceHDF H5Object

H5Object Open NDGM:FileName.h5:/DataSetName

DiceMemory MemoryObject

MemoryObject SetArray [ DiceFloat32Array ]

H5Object Read [ MemoryObject cget -this ]

set NewData [ MemoryObject GetArray ]

DiceExpr $NewData[ where( $NewData > .5 ) ] = .5

C++

Scripting ( Tcl, Perl, etc. )

Allocate 10,000 Doubles

Set Individual Values

Array Operation +, -, *, /

Array Indexing

Object Oriented Scripting Interface

Open HDF5 File, Query Number Type and Shape

Request Automatic Conversion to Float32

MemoryObject has Number Type and Shape

Clip Values to .5

DICE on HPC Resourceusing Off Screen Rendering

SSL Kerberos kinit

• Every URL Request is Intercepted to Verify Session Id

• DICE Runs as a User Process

• HTML FORMS result in Tcl Procedure Invocation

• Graphics Updated via Server-Push ( Netscape only ) or Client-Pull

• Intended for Limited Interfaces from Remote ( Slow Connection ) or Graphically Challenged Client Machines.

DICE Web InterfaceUnder Construction

NDGM

HDF5

MPI Finite VolumeMPI Finite Element

Data Mining

Steering

Visualization

Coupling

“Gently Coupling” Codes from Separate Disciplines

Finite Volume Structural Mechanics

Global Monitor

Runtime Visualization

Small Scale Analysis

Injection Flow Analysis

Distributed Data Hub

DICE and the HPC Environment

DICE Server

( User Process )

One or More Cooperating HPC

Codes

DICE User Interface

XML, HDF5 I/O

XML and Http

( SSL + Kerberos )

DICE Object Directory

COTS, GOTS and “Semi-Automatic”

Visualization

X / OpenGL