1

Mark F. AdamsH.H. Bayraktar, T.M. Keaveny, P. Papadopoulos and Atul Gupta

Ultrascalable Implicit Finite Element Analyses in Solid

Mechanics with over a Half a Billion Degrees of Freedom

(excerpts)

2

Trabecular Bone

5-mm Cube

Cortical bone

Trabecular bone

3

Micro-Computed Tomography

CT @ 22 m resolution

3D image

Mechanical TestingE, yield, ult, etc.

2.5 mm cube44 m elements

FE mesh

Methods: FE modeling

4

the vertebral body you are showing is pretty healthy from a 80 year old female and it is a T-10 that is thoracic. So it is pretty close to the mid-spine. Usually research is done from T-10 downward to the lumbar vertebral bodies. There are 12 thoracic VB's and 5 lumbar. The numbers go up as you go down.

5

1 mm slice from vertebral body

6

80 µm w/ shell

Vertebral Body With Shell

Large deformation elast. 6 load steps (3% strain) Scaled speedup

~131K dof/processor 7 to 537 million dof 4 to 292 nodes IBM SP Power3

14 of 16 procs/node

7

Athena: Parallel FE ParMetis

Parallel Mesh Partitioner (Univerisity of Minnesota)

Prometheus Multigrid Solver

FEAP Serial general purpose

FE application (University of California)

PETSc Parallel numerical

libraries (Argonne National Labs)

FE MeshInput File

Athena ParMetis

FE input file(in memory)

FE input file(in memory)

Partition to SMPs

Athena Athena ParMetis

File File File File

FEAPFEAPFEAPFEAP Material Card

Silo DBSilo DB

Silo DBSilo DB

Visit

Prometheus

PETScParMetis

METISMETISMETISMETIS

pFEAP

Computational Architecture

Olympus

8

ParMetis partitions

9

131K dof / proc.47 Teraflops - 4088 processors