grid content evolution & management

Grid Content Evolution & Management

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

Grid Content Evolution & Management:

Enabling e-Science for the Rest of Us

Thilo Ernst

Fraunhofer Institute for Computer Architecture and Software Technology (Thilo.Ernst@first.fhg.de)

Jochen Wauer

DLR Institute for Remote Sensing Technology

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

Seite 3Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

Scientific Software Just A Mouse Click Away (1)

A 1968 vision:

... Back to our hypothetical user. He seats himself at his console [...]. He dials his local computer and 'logs in' by presenting his name, problem number, and password to the monitor program. He calls for either a public program, one of his own programs, or a colleague's program that he has permission to use. The monitor links him to it, and he then communicates with that program...

- J.C.R. Licklider, "The Computer as a Communication Device" (Director, ARPA-Information Processing Techniques Office)

Did he just mean telnet?

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

Seite 4Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

e-Science – a contemporary Definition• „e-Science is about global collaboration in key areas of science and the

next generation of infrastructure that will enable it.“

• „large scale science that will increasingly be carried out through distributed global collaborations enabled by the Internet. Typically, a feature of such collaborative scientific enterprises is that they will require access to very large data collections, very large scale computing resources and high performance visualisation“

• „The Grid is an architecture proposed to bring all these issues together and make a reality of such a vision for e-Science“

• „e-Science will change the dynamic of the way science is undertaken“ (*)

(Source: UK Research Council e-Science Core Programme)

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

Seite 5Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

The entire story?

• Grid computing / e-Science still focus on high-demand, large-scale applications

• Evolution: high performance computing Grid computing e-Science renders an emphasis on such applications understandable

• But when mature Grid & e-Science technologies are available they will likely be attractive to, a wealth of smaller (and computationally less demanding) research efforts - e-Science will become ubiquitous (*)

• But for this to happen, these technologies must develop additional characteristics unrelated to the original HPC focus of the Grid movement

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

Seite 6Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

Scientific Software Just A Mouse Click Away (2)2003: Are we there?

• Internet & WWW are ubiquitious and have revolutionized the exchange of scientific information

• But progress w.r.t. scientific software lacks behind when compared with the ease of accessing scientific documents today

• „Reuse by download“: requires installation or even porting – limiting factor for broad exchange

• Distributed computing technologies exist (remote shells, CORBA, Java-RMI, Web Services) but there is no reliable, ubiquitous infrastructure to which Licklider‘s request could be addressed. Answer: No.

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

Seite 7Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

Scientific Software Just A Mouse Click Away (3)

... New question: Why?

• in much of real life science

• (non-Computer) Scientists, beyound a reasonable limit of extra effort, simply don‘t have time and/or motivation to do our (CS/IT) work:

- build, deploy, maintain e-Science-supporting platforms- integrate scientific applications into these platforms on a regular basis- and learn how to do all that in the first place

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

Scientist Computer Scientist

Seite 8Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

How did the Web succeed as a publishing medium?

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

Characteristics Enabling factors, ~time Evolutionary advantage

1. easy to use in read-mode for almost everybody

Browsers ~ 1993Search engines ~1994

more„consumers“

2. easy to write without web developer background

LaTex2HTML, MS-Office web integration ~1996

more„producers“

3. collaboration/community support

(Cross-site links ~1990) community portals ~1999

Mutual reinforcement community / medium

4. support for traditional editorial processes

content management systems ~1999

organize content growth,integrate with print media

5. infrastructure ubiquitous, commoditized, reliable

ISPs/hosters become an industry (~1998)

Specialization / division of work

Seite 9Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

Towards „Broad e-Science“ – what is missing?

For e-Science/Grid technology to achieve the role of the primary collaboration medium in „broad scientific computing“, it seems desirable to develop similar „platform survival characteristics“:

1. Ease of use for passive users („consumers“) – Grid/Science portals / Desktops

2. Ease of use for „producers“ (authors of services) - ?

3. Collaboration support – at the core of the Grid / e-science movement

4. Support for editorial processes - ?

5. Infrastructure gets ubiquitous & commoditized – Industry interest

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

Seite 10Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

DLR VirtualLab: an e-Science precursor (1)- technology transfer project 2000-02 („50%“)

- make scientific software accessible from every web browser (like scientific documents already are)

- tap unexploited application potential

- promote technology transfer as well as scientific exchange (internally, too!)

- make better use of this part of DLR‘s scientifíc output

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

Seite 11Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

DLR VirtualLab: an e-Science precursor (2)

- online at http://vl.nz.dlr.de

- Pre-Grid (dedicated cluster)

- Zope/Python,OpenPBS, mySQL, OpenLDAP, ...

- a specialized ASP platform for an „interesting“ class of scientific software

- IEEE CiSE, Feb. 2003

- project now continues as a DLR/Fraunhofer collaboration

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

Seite 12Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

DLR VirtualLab: Concepts for broad adoption of e-Science(1)Care about the process

- keep the barrier to entry low both for „consumers“ and for „producers“

- learn from publishing (traditional & web-based)

- use staging !

Administrator Anonymousexternal user

App

lica

tion

Ser

ver

Author

I N

TE

RN

ET

Integrator

Internal user

registeredexternal user

External users

Repository

ScientificPrograms

I N

TR

AN

ET

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

Seite 13Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

DLR VirtualLab: Concepts for broad adoption of e-Science(2)Make life easy for consumers &

producers

- don‘t exclude important classes of users

- „black-box“ integration: no internal modifications whatsoever

- Provide „cheap“ UIs through generation

- strive to re-use existing (G)UIs

- Integration tools, perhaps web-based

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

Seite 14Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

DLR VirtualLab: Concepts for broad adoption of e-Science(2)Make life easy for consumers &

producers

- don‘t exclude important classes of users

- „black-box“ integration: no internal modifications whatsoever

- Provide „cheap“ UIs through generation

- strive to re-use existing (G)UIs

- Integration tools, perhaps web-based

ID(na='Lambda', # Name

pe='_NF and _Lambda_use',# precondition

ty='FloatType', # type

de='0.5', # default value

ce='Lambda > 0.0', # constraint

un='mu_m', # physical unit

an='Input wavelength') # annotation

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

Seite 15Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

DLR VirtualLab: Concepts for broad adoption of e-Science(2)Make life easy for consumers &

producers

- don‘t exclude important classes of users

- „black-box“ integration: no internal modifications whatsoever

- Provide „cheap“ UIs through generation

- strive to re-use existing (G)UIs

- Integration tools, perhaps web-based

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

Seite 16Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

DLR VirtualLab: Concepts for broad adoption of e-Science(3)Integrate human-readable information with executable (software) resources

- online help and background info

- (computational) „service search engines“ for managing the new e-Science „resource deluge“

-Structured metadata are good, but documentation can serve as a good fallback „fingerprint“

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

Seite 17Arc

hiv

ieru

ng

san

gab

en

Institute for Remote Sensing Technology

Innovation Management and Technology Marketing

Conclusions• e-Science will likely evolve to broad/ubiquitous e-Science

• support not only HPC users but the broader/general scientifc computing community

• to help that process, make life easy for

A widely deployed Grid-based e-science infrastructure is a publshing medium- support the process by content management, leveraging suitable technologies

Grid Content Evolution & Management - e-Science For The Rest Of Us T.Ernst, J.Wauer

„ Grid content producers“

- strive for allowing black-box integration of apps & (G)UIs - provide integration tools

„ Grid content consumers“

- don‘t exclude important groups - simple browser-based access for occasional users remains valuable