accelerating data-intensive science by outsourcing the mundane

www.ci.anl.govwww.ci.uchicago.edu

Accelerating data-intensive scienceby outsourcing the mundane

Ian Foster

Alfred North Whitehead (1911)

Civilization advances by extending the number of important operations which we can perform

without thinking about them

J.C.R. Licklider reflects on thinking (1960)

About 85 per cent of my “thinking” time was spent getting into a position to think, to make a decision, to learn something I needed to know

For example … (Licklider again) At one point, it was necessary to compare six

experimental determinations of a function relating speech-intelligibilityto speech-to-noise ratio. No two experimenters had used the same definition or measure of speech-to-noise ratio. Several hours of calculating were required to get the data into comparable form. When they were in comparable form, it took only a few seconds to determine what I needed to know.

Publish results

Collectdata

Design experiment

Test hypotheses

Hypothesize explanation

Identify patterns

Analyzedata

Research hasn’t changed much in 300 years

Pose question

Discovery 1960: Data collection dominates

Janet Rowley: chromosome translocations

and cancer

800,000,000,000 bases/day30,000,000,000,000 bases/year

Discovery 2010: Data overflows

Meanwhile, we drown in administrivia

The Federal Demonstration Partnership’s faculty burden survey

You can run a company from a coffee shop

Software

Platform

Infrastructure

Salesforce.com, Google,Animoto, …, …, caBIG,TeraGrid gateways

Varieties of “* as a Service” (*aaS)

Software

Platform

Infrastructure Amazon, GoGrid,Microsoft, Flexiscale, …

Varieties of * as a service (*aaS)

Software

Platform

Infrastructure Amazon, GoGrid,Microsoft, Flexiscale, …

Google, Microsoft, Amazon, …

Varieties of * as a service (*aaS)

Perform important tasks without thinking

Web presence Email (hosted Exchange) Calendar Telephony (hosted VOIP) Human resources and payroll Accounting Customer relationship mgmt Data analytics Content distribution IaaS

Perform important tasks without thinking

Web presence Email (hosted Exchange) Calendar Telephony (hosted VOIP) Human resources and payroll Accounting Customer relationship mgmt Data analytics Content distribution

What about small and medium labs?

Research IT is a growing burden

Big projects can build sophisticated solutions to IT problems

Small labs and collaborations have problems with both

They need solutions, not toolkits—ideally outsourced solutions

Medium science: Dark Energy Survey

• Every night, they receive 100,000 files in Illinois

• They transmit these files to Texas for analysis (35 msec latency)

• Then move the results back to Illinois

• This whole process must run reliably & routinely

Image credit: Roger Smith/NOAO/AURA/NSF

Blanco 4m on Cerro Tololo

Open transfer sockets vs. time

[Image: Don Petravick, NCSA]

A new approach to research IT

Goal: Accelerate discovery and innovation worldwide by providing research IT as a service

Leverage software-as-a-service (SaaS) to• provide millions of researchers with

unprecedented access to powerful research tools, and

• enable a massive shortening of cycle times intime-consuming research processes

Time-consuming tasks in science

• Run experiments• Collect data• Manage data• Move data• Acquire computers• Analyze data• Run simulations• Compare experiment

with simulation• Search the literature

• Communicate with colleagues

• Publish papers• Find, configure, install

relevant software• Find, access, analyze

relevant data• Order supplies• Write proposals• Write reports• …

Time-consuming tasks in science

• Run experiments• Collect data• Manage data• Move data• Acquire computers• Analyze data• Run simulations• Compare experiment

with simulation• Search the literature

• Communicate with colleagues

• Publish papers• Find, configure, install

relevant software• Find, access, analyze

relevant data• Order supplies• Write proposals• Write reports• …

Discover endpoints, determine available protocols, negotiate firewalls, configure software,

manage space, determine required credentials, configure protocols, detect and respond to failures, determine expected performance, determine actual performance, identify diagnose and correct network misconfigurations, integrate with file systems, …

Data movement can be surprisingly difficult

Grid (aka federation) as a service

Globus ToolkitBuild the Grid

Components for building custom grid solutions

globustoolkit.org

Globus OnlineUse the Grid

Cloud-hostedfile transfer service

globusonline.org

Globus Online’s Web 2.0 architecture

Fire-and-forget data movementMany files and lots of dataCredential managementPerformance optimizationExpert operations and monitoring

Web interface

HTTP REST interfacePOST https://transfer.api.globusonline.org/ v0.10/transfer <transfer-doc>

Command line interfacels alcf#dtn:/scp alcf#dtn:/myfile \ nersc#dtn:/myfile

GridFTP serversFTP servers

High-performancedata transfer nodes

Globus Connecton local computers

Globus Connect to/from your laptop

Almost always faster than other methods

gogucscptunedguc

0.001 0.01 0.1 1 10 100 1000Megabyte/fileArgonne NERSC

Monitoring provides deep visibility

Globus Online runs on the cloud

Data movers scale well on Amazon

11 x 125 files200 MB each

11 users12 sites

SaaS facilitates troubleshooting

Moving 586 Terabytes in two weeks

NSF XSEDE architecture incorporatesGlobus Toolkit and Globus Online

Publish results

Collectdata

Design experiment

Test hypotheses

Hypothesize explanation

Identify patterns

Analyzedata

Next steps: Outsource additional activities

Pose question

A use case for the next steps

• Medical image data is acquired at multiple sites• Uploaded to a commercial cloud• Quality control algorithms applied• Anonymization procedures applied• Metadata extracted and stored• Access granted to clinical trial team• Interactive access and analysis• More metadata generated and stored• Access granted to subset of data for education

Required building blocks

• Group management for data sharing– Scheduled September, 2011, for BIRN biomedical

• Metadata management– Create, update, query a hosted metadata catalog

• Data publication workflows– Data movement, naming, metadata operations, etc.

• Cloud storage access– And HTTP, WebDAV, SRM, iRODS, …

• Computation on shared data– E.g., via Galaxy workflow system

www.globusoline.org

Summary

• To accelerate discovery, automate the mundane

• Data-intensive computing is particularly full of mundane tasks

• Outsourcing complexity to SaaS providers is a promising route to automation

• Globus Online is an early experiment in SaaS for science

For more information

• Foster, I. Globus Online: Accelerating and democratizing science through cloud-based services. IEEE Internet Computing(May/June):70-73, 2011.

• Allen, B., Bresnahan, J., Childers, L., Foster, I., Kandaswamy, G., Kettimuthu, R., Kordas, J., Link, M., Martin, S., Pickett, K. and Tuecke, S. Globus Online: Radical Simplification of Data Movement via SaaS. Preprint CI-PP-05-0611, Computation Institute, 2011.

Thank you!

foster@anl.govfoster@uchicago.edu

accelerating data-intensive science by outsourcing the mundane

relevant data

data collection

data overflows8

research itgoal

important tasks

powerful research tools

thinking time

voip human resources

Technology

customer success - r systems...offshore outsourcing started...

101 mundane treasures

practical mundane astrology

the art of the mundane

introduction - university of...

making the mundane beautiful

finance shared services and outsourcing - ibm · pdf...

glory in the mundane

mundane or national astrology

magic made mundane 101213 top -...

information architecture of the mundane

mundane 2 marvelous

accelerating success is humanly possible€¦ · service...

adventures of the mundane - sainsbury centre€¦ ·...

mundane astrology book

infosys 3.0 | accelerating growth | annual report 2011-12...

creatures mundane & supernatural

mundane and everyday examples

israelite bread: mundane and consecrated

mundane astrology - aoa.co