Networked Data Storage and Analysis for the Wisconsin Regional Materials Network

6/3/2015 M&M 2015 Portland 1

Jon J. McCarthy, Paul M. Voyles, Julie A. Last, Davis R. Bittner, Ian W. Sadkovich

Wisconsin Materials Institute, University of Wisconsin, Madison WI, USA Materials Science and Engineering Department, University of Wisconsin, Madison WI, USA

Materials Science Center, University of Wisconsin, Madison WI, USAComputer-Aided Engineering Department, University of Wisconsin, Madison WI, USA

Our big data issues at UW Madison

Instrument data output is growing. FAST!

Issues with continuing to transfer data the Same Old Way

The importance of including off the local computer Analysis Capability

A couple examples

Things we have learned so far

M&M 2015 Portland 26/3/2015

Problem Statement:Large Images/multispectral data sets Need to be storedOFF the Instrument COMPUTER

Virus avoidance is critical Dedicated sub networks How Large is large? EM Tomography data

Now 0.5 Gb per 100 slice stack, soon to be 2 to 16+ GB per stack

But also Processed . Which leads to even more imagesand more storage space Often Proprietary commercial software required.

How many licenses can you afford?

Can your collaborators access data and analysis it output easily? Without access to rest of your computer networks and restricted software?

M&M 2015 Portland 36/3/2015

Wi Regional Materials Network

EM Tomography Data Storage Requirements: Electron Tomography of Cryo Sections :

What is Next?Happening now in fact!

6/3/2015 M&M 2015 Portland 4

The Output is 0.5 GB of raw Image data in a single tilt series.

With no binning its 2 GB of data.

K2 16 MegapixelDirect Electron Detector

The Output is potential for 4 to 32 GB for a single tilt series!

GIF: binned 2X2

AP Tomography Data Storage Requirements:

6/3/2015 M&M 2015 Portland 5

The Output is 2.8 GB of raw Image hit data file that is processed to generate this image.

Getting data off the instrument: No Portable Drives!The trouble with Thumb and Portable Drives: Viruses and Limited Capacity

M&M 2015 Portland 66/3/2015

NOT Acceptable, but all to common!

Getting data off the instrument: Limited LANInternal sub-net to another Lab PC for Storage; Lock rack door to the both Instrument PCs!

M&M 2015 Portland 76/3/2015

Instrument Control PC

Support PC(crossover Cable)

This was a satisfactory solution for several years

BUTissues are: LAN running out of disk Storage

20 users, output is 4 TB per year. Limited LAN host is maxed out below that level!

Duplicating hosts and more storage and Licensed processing software quickly becomes cost prohibited !

SO users bring out the portable drives AGAIN

. OR

Users must come to the host to process data. One user at a time can process data (its

not a server, just a host) Access for outside UW Collaborators very

difficult!

We needed a scalable Solution!

Networked Data and Analysis SolutionA data storage and processing on a secure, networked platform:

We chose the Citrix XenAPP Platform

M&M 2015 Portland 86/3/2015

Citrix XenAPP provides delivery over the Internet of data analysis applications with access to a central data repository. Users acquire their data on the instrument. Data is copied off the instrument

computers to a centralized data store. The data store is connected to an analysis

software server. Users can connect to the analysis software

server from any Windows or Mac computer with Internet connection faster than 1 Mbit/sec.

Raw data and analysis Output are stored on the server, or copied back to the client computer.

NDA Advantages The Instrument control computers do not see Thumb Drives, USB Storage, or even the UW Network! Secure Login, accesses only client machine and the XenAPP server. Hardware easy to expand as storage needs grow; Currently, 24 cores on two processors and 256 MB of RAM, 4 TB of storage. EASY to SCALE! Data can be sent to server directly from instrument Support Computer and downloaded to client computer as needed. The XenApp client runs on the users computer, but the application runs on a University (CAE) managed Windows server. XenApp delivery is legal for most end user license agreements (EULAs) for data analysis applications. Outside users cannot access other UW resources or other computers on the CoE network.

6/3/2015 M&M 2015 Portland 9

Example: Launching XenAPP

6/3/2015 M&M 2015 Portland 10

Secure Login

Access to only these applications

Client and Server Drives auto-mapped

6/3/2015 M&M 2015 Portland 11

Client drives

XenAPP Server Drives

NO Access to UW computers

Example: EM Tomography

6/3/2015 M&M 2015 Portland 12

On my desktop but running on XenAPP Server

Tomogram created with IMOD/etomo on XenAPP! Made into an AVI with Fiji

Frame from a FIJI movie of a rendered Tomogram

6/3/2015 M&M 2015 Portland 13

Movie of FHV rendered stack running in XenAPP On my laptop (i7,

8GB Dell) At home, 11

Mbps download speed!

Collaborator Access Example

6/3/2015 M&M 2015 Portland 14

UW Milwaukee/UW Madison/Thermo Scientific Collaborators

Li Cells, Built, ChargedIn Milwaukee.

Anodes extracted in Glove Box, Loaded into Thermo Vacuum Transfer Vessel in Madison

XPSSpectralImaging

Data

Data on Instrument Computer transferred to NDA Server

Collaborator Access Example

6/3/2015 M&M 2015 Portland 15

Spectral Images Processed, Quantified entirely on Thermo Avantage on XenAPP Server

Atomic % for all elements from three anodes determined,and plotted in Excel, by collaborators in MKE and Madison!

TIPS: What we have Learned so far Unless you can manage the Server yourself, get a key contact in your IT group to be

the owner of XenAPP administration. Admin and negotiation of License and EULA How do you load and configure plugs or upgrades?

Commercial Software seems to be more likely to be easy to configure on a LAN and XenAPPo Many packages (with a LAN License) work out of the boxo Shareware is often configured for a single user and a desktop/laptop

Often use code or scripts that select specific files or path names that may be blocked on the Server! It took three rounds to get 3DMOD to work, then 3 more to get etomo to work

Have a Skilled User do testing of any new XenAPP application! IT folks are computer savvy, but not experts on your application! You dont want to learn the software and debug XenAPP install issues at the same time! Have the tester user write up how to use IMOD on XenAPP tutorial document.

6/3/2015 M&M 2015 Portland 16

Acknowledgements

6/3/2015 M&M 2015 Portland 17

Co-AuthorsPaul M. VoylesMaterials Science and Engineering Department

Julie Last, Alex Kvit and John JacobsMaterials Science Center

Davis R. Bittner and Ian W. SadkovichComputer-Aided Engineering Department

Special Thanks (for tomography work) to:Ken J. ErtelFellow,Institute For Molecular VirologyUW Madison

Desiree BenefieldPost Doc Research AssociateMorgridge Institute for Research

Weixin Chen, Research AssociateMaterials Science and Engineering DepartmentUW Madison

External Collaborators:Xingkang HuangMechanical Engineering Department University of Wisconsin-Milwaukee

Bill SgammatoThermo Fisher Scientific

6/3/2015 M&M 2015 Portland 18

QUESTIONS?

Movie is so Cool I have to show you!

6/3/2015 M&M 2015 Portland 19

6/3/2015 M&M 2015 Portland 20

Motivation for moving from Limited LAN to NDA System

6/3/2015 M&M 2015 Portland 21

Data Output and Processing bottlenecks: Quickly running out of disk spaceif one user produces 16.5 GB in a week, and you have 20

user-> 4 TB per year. Limited LAN host is maxed out below that level. The data output will quadruple with the direct electron detector!

Duplicating hosts and more storage and Licensed processing software quickly becomes costprohibited !

Users must come to the host to process data. Only one user at a time can process data (its not a server, just a host)

It often takes as long (or longer!) to process tomograms as to collect the data! If Host is in the lab (Atom Probe) , it is cold and noisy and crowded. The next user is collecting their

data, space is crowded!

SO what Happens? Users get a license for their PC and use the Portable Drives again to move data to their own

machine. Multiple licenses for software raise cost for each user group, so still have the one person in the

lab processing at a time!

22

Materials Science Center (MSC) Three FESEMs and AURIGA FIB, PHI Auger, 4 XRDs, including Bruker Discover D8 (microfocus

source, automated stage, WAXs detector) JY Horiba Micro-imaging Raman and Micro FT-IR, UV/VIS/NIR spectroscopy, DI Multimode AFM, Bruker Catalyst Bio AFM, Zeiss META CLSM,Thermo Imaging XPS, FEI 200KV Titan, FEI TF-30 CryoTEM for electron tomography, Cameca LEAP 3000si Atom Probe.

Soft Materials Laboratory (SML) Rheometer, Dynamic Mechanical Analysis, 2 HPLC, TGA, DSC, contact angle, Ellipsometry,

Zeiss 1550 VPSEM, Thermo Scientific Micro-Imaging Raman

Wisconsin Center for Advanced Microelectronics (WCAM) Seven Bay, 10,000 Square foot clean room Facility, over 60 fab tools CMOS devices, MEMS, soft lithography, DC/RF sputter deposition tool.

UW College Of Engineering Shared Facilities

Instruments in Blue (most recent!) create Analytical Spectral Images and Image stacks -which brought us to a tipping point, but Electron Tomography pushed us over the edge!

Loading an IMOD stack from client HD

6/3/2015 M&M 2015 Portland 23

Getting data off the instrument: Solution #2

Local 1 to 2 TB portable hard disk

It stays in the instrument room - always!

But does It really??

M&M 2015 Portland 246/3/2015

ISSUES: Researchers must come here to

Process It often takes as long (or longer!) to

process tomograms as to collect the data!

Only one user at a time! For the Atom Probe;

Its cold and noisy in the lab The next user is collecting their

data, space is crowded!

NEED to Process the data somewhere ELSE!

Getting data off the instrument: Limited LANA support computer, with crossover Ethernet cable to instrument control computer!

M&M 2015 Portland 256/3/2015

Instrument Control PC

Support PC(crossover Cable)

ISSUES: Better than solution 2, but one thumb

drive inserted in the wrong computer brought down our Titan! Led to a Disk Rebuild!

It often takes as long (or longer!) to process tomograms as to collect the data!

Only one user at a time Run out of disk space on the support

Computerif one user generates 16.5 GB, and you have 20 users

NEED to process Data somewhere ELSE

Support computer monitor

Networked Data Storage and Analysis for the Wisconsin Regional Materials NetworkOur big data issues at UW MadisonProblem Statement:EM Tomography Data Storage Requirements:AP Tomography Data Storage Requirements:Getting data off the instrument: No Portable Drives!Getting data off the instrument: Limited LANNetworked Data and Analysis SolutionNDA AdvantagesExample: Launching XenAPPClient and Server Drives auto-mapped Example: EM Tomography Frame from a FIJI movie of a rendered TomogramCollaborator Access ExampleCollaborator Access ExampleTIPS: What we have Learned so farAcknowledgementsSlide Number 18Movie is so Cool I have to show you!Slide Number 20Motivation for moving from Limited LAN to NDA System Slide Number 22Loading an IMOD stack from client HDGetting data off the instrument: Solution #2Getting data off the instrument: Limited LAN