Development Timelines

Ken Kennedy

Andrew ChienKeith Cooper

Ian FosterJohn Mellor-Curmmey

Dan Reed

What We Have Today

• Prototype Testbeds with middleware

• Prototype of execution components in ScaLAPACK and Cactus

• Design for the execution environment—Implementation of resource specification (AART)—Interfaces for Scheduler/Resource Negotiator

– Prototype resource scheduler (Dail)– Prototype renegotiator (Sievert)

—Prototype Contract Monitoring System

• Grid ready libraries with performance models

• Tools for extracting information from program executables—For performance estimation on single nodes

• DSL design for Signal/Image Processing

What We Will Have on CGrADS Day One

• Prototype Execution System for Heterogeneous Clusters—Prototype Scheduler/Resource Negotiator—Binder—Run-time system—Contract monitor (working together)

• ScaLAPACK re-implemented using the prototype

• Cactus using generic resource selector

• Prototype automatic performance modeling for black boxes

• Script-based application composition (without optimization)

• Testbeds—Microgrid running ScaLAPACK and Cactus—Integrated NWS and prediction in MacroGrid and

configuration tools

Currently Targeted Application Milestones

• Three codes concurrently, each engaged for two-three years

• 2002—Cactus: traditional PDE solver, aggressive application scenarios—CAPS: dynamic data acquisition and real-time data ingest—ChemEng Workbench: application service scenarios prototypes

• 2003—Cactus: by now transitioned to operational use by application

group—CAPS: adaptive execution for high-speed prediction—ChemEng Workbench: application service scenarios operational—CMS/GriPhyN: query estimation and dynamic scheduling—BIRN-like distributed bioscience: emergent behavior issues

• 2004—CAPS: by now transitioned to operational use by application group—CMS/GriPhyN: large-scale experimentation in production settings—NEES: application service and real-time data analysis scenarios

NCSA Linux Cluster

5) Secondary reports complete to master

Master Condor job running at

Caltech

7) GridFTP fetches data from UniTree

NCSA UniTree - GridFTP-enabled FTP server

4) 100 data files transferred via GridFTP, ~ 1 GB each

Secondary Condor job on

WI pool

3) 100 Monte Carlo jobs on Wisconsin Condor pool

2) Launch secondary job on WI pool; input files via Globus GASS

Caltech workstation

6) Master starts reconstruction jobs via Globus jobmanager on cluster

8) Processed objectivity database stored to UniTree

9) Reconstruction job reports complete to master

CMS Data Reconstruction Example

Scott Koranda, Miron Livny, others

Program Preparation System Milestones

2002—Preliminary automated support for performance models

(black boxes)—Binder includes local optimization, inserts probes and

actuators—Prototype DSL for signal processing—Evaluate original COP design

2003—Binder support for global optimization—Experiment with contract monitoring/reporting in

applications—Evaluate and extend DSL support for signal processing

2004—First dynamic optimizer prototype, plan for retargeting

Binder—Initial telescoping language prototype based on experience

with signal processing DSL—Whole Program Compiler generates initial COPs

Program Execution System Milestones

• 2002—Virtual organization management tools—Resource selector and application manager prototypes—Temporal contract violation specification—Rescheduling models including distribution costs & info

quality

• 2003—Integrated resource monitoring and prediction prototype—Resource selector and application manager tools—Composable contract specification and tools—Reconfigurable object program specification—Scheduling models for highly parallel and data Grid

applications

• 2004—Enhanced application and resource measurement

infrastructure—Enhanced resource scheduling infrastructure with adv.

resv., etc.—Performance economics for global resource scheduling

Execution Environment Milestones

GrADSoftv1

GrADSoftv2

GrADSoftv3

Dist. Comp.:Cactus

App Server:ChemEng

Real-time:CAPS

Data Grid:CMS

Real-time:CAPS

Data Grid:CMS

AutoPilotGlobus

MPICH-GNWS

NetSolveAppLeS

Cactus

Virt. OrgMgmt Tools

ResourceSelector

ApplnManager

ContractViolation

ChemEng

MacroGridMicroGrid

Year 1-2 Year 2-3 Year 3-4

ComposableContracts

SchedulingModels

ReconfigurableObject

Programs

ResourceMonitoring

App Server:NEES

PerformanceEconomies

EnhancedResource Mgt

ResourceMonitoring

Software Development Process

Andrew A. ChienDepartment of Computer Science and Engineering

University of California, San Diego

http://hipersoft.rice.edu/stc_site_visit/talks/SWProcess.pdf

Software Development Process

• Development Efficiency versus Robust Output—Key is choosing where to live on this spectrum

• CGrADS Objectives—Ground breaking research—Rapid development of functionality and experimentation—Difficult to know what will become widely used

– Hardening of selected components

Software Development Approach

• Experimental development with lightweight process

• Core development with close to full process

• Targeted hardening of what becomes core—Enable researchers inside and outside CGrADS to build on

core—Software output not supported as a product

Software Development Process

• Project Software Manager defines and enforces process; drives progress

• Infrastructure—Revision control system (CVS,SourceSafe,Clearcase)

– Code, documents—Defect tracking system (ClearQuest)—Design, documentation, coding guidelines—Software infrastructure

• Process—Lightweight Process—Industrial Process

Two Flavors of Process

• “Lightweight” Process (Research Strength)—Requirements and Design; Review—Implementation and Test; Review—Integration; Test—Iterative Improvement

• Full Process (Industrial Strength)—Requirements; Review—Design; Review—Implementation; Test and Review—Integration; Test—Iterative Improvement