1

Out-of core Streamline Generation Using Flow-guided File Layout

Chun-Ming Chen

788 Project

2

Background

• Visualize flow fields with streamlines • Scientific data is huge – Traditional: Compute in clusters– Drawbacks: • High Equipment Cost• Inter-node communication

http://vidi.cs.ucdavis.edu/publicationshttp://nuweb2.neu.edu/

3

Background

• Nowadays: multi-core CPU on single machine• May not have enough memory capacity• Out-of-core computation is needed– Out-of-core: data cannot be fully loaded into main

memory

4

Goal

Compute streamlines on a lower-cost multi-core machine with limited memory, given arbitrary seeds

5

Demand Paging Algorithm

• Preparation Stage:– Break flow fields into blocks

• Streamline Generation Stage:– Only load needed blocks during computation– Release least recently used (LRU) block when

memory full

Load data from Disk Compute Release data

(LRU)

Store data in memory

pool

6

Multi-core streamline computation

Threaded Computation

Seeds for block 1

Seeds for block 2

Seeds for block 3

Seeds for block 4

Threaded Computation

New seeds generated from

block 1

Job Queue

7

Problem of Out-of-core Computation

• Earlier tests: 1Gb Data– Environment:• 8-core Intel Machine• Limit 25Mb memory usage

– Time Generating streamlines: 8.436 s– Time Loading flow field : 32.524 s

• IO is the bottle neck

8

More tests

• Read all blocks in a 6Gb data• Unit block size: float 16x16x16 (49152 bytes)

Total 131,072 blocks– Random access: 249.146794 sec– Sequential read: 13.594309

sec– Reverse-Sequential read: 264.262752 sec

• Sequential read can be 20 times faster• Reason: Disk Prefetching

9

File Layout

• Re-arrange data to increase more sequential reads

• Hilbert Curve Layout:

10

Result of Scheduling for Hilbert Curve Layout

• Scheduler: only read forward• Test: 1Gb Data– Environment:

• 8-core Intel Machine• Limit 25Mb memory usage

• Old test:– Time Generating streamlines: 8.436 s– Time Loading flow field : 32.524 s

• Hilbert layout:– Time Generating streamlines: 7.890 s– Time Loading flow field : 0.610 s

11

Layout By Flow Direction

12

Next And Conclusion

• Next:– Better layout?– Re-arrange data based on flow direction– NP-hard Problem

• Conclusion:– If we want to analyze large scientific data in a single

machine, out-of-core computation is required now and also in the future

– Good File layout is important for out-of-core computation