grid market - scut market dr simon c.w. see director grid and high performance computing technology...

Grid MarketDr Simon C.W. SeeDirectorGrid and High Performance Computing TechnologySun Microsystems Inc.

Sun Proprietary/Confidential: Internal Use Only

Grid Economic or Grid Market

•Motivation>to make grid to be the mainstream computing model,


“Utility” & “Grid” Definitions

A computing infrastructure that someone else owns and manages. You access this infrastructure only whenyou need to, pay only what for you use, and have nolong-term commitment.

A specific way systems are linked in a horizontalfashion. Sun is offering a grid infrastructure (or gridfarm) for customers to access on a pay-as-you-gobasis (versus a high-end, mission-critical data center.)

UTILITY

GRID


Grid (Background)• Current Practice of Grid

> voluntary sharing of compute resources,> establishing small-scale grid dedicated to a

specific purpose> FREE !!!?????

• BUT > this is simply not sustainable for global level

generic grid > WHO is paying for it?

•Footnote position, 12 pts.


Grid Market• Approach

> an efficient supporting platform and mechanism to encourage> resource owners to issue their idle resources > customers can get the satisfying resources.

> Using markets in Grid computing to commercialize their grid resources


Definition• Traditional Market

> a social arrangement that allows buyers and sellers to discover information and carry out a voluntary exchange.

• Grid Market > a software platform with the business

mechanisms to support trading between grid users.

e-Service Market ConceptA Market Place and Exchange provides economic incentive to trade

“Resource Owners”

1. Pricing Model2. Catalog 3. Billing Support –

Trading Agents Accounting Agents

1. Utility Model – The Contract

2. Broker to fulfill the contract

“Resource Users”

Enablers to Trade Enablers to Trade

Exchange

Broker

Broker

Broker

UserUser

User

User

ServiceProvider

User

ServiceProvider

ServiceProvider

Broker

Resource Provider

ServiceProvider

E-Service Market Place


Market Advantage• Encourages more users to adopt Grid technology,

especially in enterprises.• Provides incentives for resource owners to provide their

idle resources, which is helpful for establishing large-scale, mature grid system.

• Enforces efficient utilization of grid resources> buyers who value a resource most highly will buy from sellers

most willing to sell. > Provide access for even small businesses to temporary grid

resources ( too expensive to acquire on their own, or just to meet their short term peak demand).

• Both buyer and seller can easily design their trading policies based on their current status so as to maximize resources’ utilization and their benefits.


Industry Grid or Utility• Sun Grid Compute Utility,• IBM’s On-demand Computing (ODC), • Platform Enterprise Grid Orchestrator (EGO),• HP Adaptive Enterprise


IBM’s On-demand computing (ODC (1) Autonomic computing.• (2) Fast response to external business-affecting

changes. • (3) Adaptive business processes to protect

revenues and contain costs. • (4) Complex interactions inside and outside of

organizational boundaries. • (5) Resilience against external threats such as

viruses, intrusions, and power outages


On Demand Environment Arch.


Platform Enterprise Grid Orchestrator (EGO)


HP Adaptive Enterprise

• Agile, Adaptive• Technology

> Tycoon


Sun Grid: Worlds Only Grid Based Compute Utility

Technical Complexity Managed by Sun and Hidden from ViewProvides Simple Plug-and-Play Experience for End Users

Easy, Secure, Affordable, Powerful

• Standard Infrastructure• Reliable• Scalable• Predictable pricing• Metered pricing• Multi-source• Multi-tenancyNetwork.com


Sun Grid (Co-op ready)


The Face of Utility ComputingEasy Access to Compute Resource to Anyone on the Network


Customer ChoiceBuy the System, Consume the Utility, or Both

Sun Grid Private Utility

Commercial Sun Grid

Public Sun Grid

Consume it:Sun's Utility PortfolioBuy it:

Sun's Hardware & Software

Common Technology and IPCommon Architecture and DesignCommon Building Blocks


The Evolution of Public Sun Grid Utility

Phase 3

InfrastructurePlatform

Services Platform

2004 2005 2006

Continuous Innovation

Continuous Innovation

2007

Portal AccessHPC and Batch Apps

Programmatic AccessMiddleware ServicesISV interfaces

Multi sourceService Provider Packaging

2008

Service Aggregation


What Can Customers Do Today?Expedite success running compute intensive batch jobs

Digital Movie RenderingQuantum Chromodynamics

Radiation Cancer TherapyOffload Peak Traffic Flow

Stellar Evolution

Econometrics

Dow-Jones Forecasting

Oil Well Exploration

Very Large Scale Integration (VLSI) Design

CAE Military Simulation ModelingThe Synthetic Environment

Digitalization & Network ModelingSimulation Training

Monte Carlo Analysis

Molecular Drug Testing

Thousands of subscribers!

> 1,000 Developers Registered!


GESA-WG (Closed)• an open extensible economic service

architecture (based around OGSA Grid Services) to enable the provision and consumption of Grid Services.

• a Grid Banking Service (GBS) that records these financial transactions.

• Chargeable Grid Service (CGS) that encapsulates an existing Grid Services with the functionality necessary for it to be purchased.

• Enable a variety of economic models with these services but not to explore the economic models themselves.

●


GRACE (Gridbus)


GRACE (Gridbus)• Grid Resource Broker (e.g., Nimrod/G) • Grid Resource and Market Information Server

(provided by Globus MDS and extended services for resource access price)

• Grid Open Trading Protocols and API


The Computational Markets ●funded under the DTI e-Science Core Technology programe


The Computational Markets


Compute Power Market (CPM)

• Repository of information on providers

• Agents for consumers and providers

• Mechanisms for updating the information

• Interaction with other markets


Businss Grid (Japan)● Increasing the efficiency of in-house IT systems● Managing expected load increases● Enhancing the availability of the e-Japan system● Features of the Business Grid middleware are:● All information relating to a business application can be described

and retained in a defined format.● The IT resources used by business applications are virtualized as

hosting environments.● business applications can be deployed automatically on distributed

IT resources● a business application operating on an IT resource can be moved

to another resource or more resources can be added to a business application.


Mapping System Attributes


Business and Pricing Models

• Commodity Market Model• Posted Price Model• Bargaining Model• Tendering Model• Auction model


• Time Mode> Instant> Subscription> Agreement

• Trading Object thru Proxy> Commission Proxy> Confirmation Proxy> Non-Porxy


Dynamic Pricing• the terms on which the trading objects ( products or

services) are exchanged. • a reflection of the current state of the market• Price Dispersion

> Spatial and/or temporal• Price Discrimination

> 1st Degree – Perfect Differentiation> 2nd Degree – Nonlinear Pricing> 3rd Degree - the producer sells products to different

people for different prices, but every unit of product sold to a given person sells for the same price.


Model of Dynamic pricing• Is an optimization problem• Type of Model

> Inventory-based Model> Data-driven Model

> e.g Customer-based Model> Auction-based Model> Game Theory Model> Machine-based Model


Research Work• Customer Behavior Mode

> SunLab• Re-enforce Learning• Trading Grid Services

> University of Karlsruhe• Grid Market Framework

> NTU, APSTC• SORMA

> 8 Univ + Sun(APSTC) + 3 Commericial companies• Simulation Technology

> HUST + APSTC

Grid Market Framework

Grid Exchange, Grid Market, etc...

Grid Middleware (Globus, UNICORE…)

Grid Service / Grid Resources / Grid Applications

Data Grid Computational Grid Visualisation Grid …...

Business Development Toolkit

Workflow Engine Data Engine Visual Engine

Pricing Formation Charging Tools

Registry management

Identity management

Negotiation management

License mangement

Business Management Toolkit

Billing Accounting Banking

Trading management

Contract management

Service Settlement

Secure Trust Model Reporting Metering

Business Support Toolkit

Loyalty management

Monitoring Toolkit

Service Routing Portal Tool

…...

Customer layer

Grid Market Layer

Product Layer

Grid Fabric

Customer Toolkit

Decision Support Search Engine Service Broker ...


Grid Service Market Framework Prototype v1.0 Portal

Portal

Contract Mgmt Publish Payment Search

Searcher Agent

Relational Database

Trader Agent

Authentication Sign In/Out

XML Database



Banker Agent


The Problem...

“Only 5% of the applications are today suitable for unfettered

deployment to any utility computing environment...”

How do we accelerate the development of a library of

suitable applications?


Categorizing WorkloadsData Flow Defines System Requirements

Application DevelopmentCompute GridWorkgroup

Electronic Design SimulationStructural AnalysisEAI Servers

J2EE Application ServersDirectory

Security

Streaming Media

Web Serving

Proxy Caching

Thermodynamics

Seismic Analysis,Reservoir Modeling

File ServerOLTP Database

Nuclear Simulation/Weapons Modeling

Data WarehousingMeteorology/Climate Simulation

Financial Risk/Portfolio AnalysisMonte Carlo Simulation

Genomics, Cheminformatics

Batch

WebData

HPCApplication

Data Analysis

Compute

NetworkIntensive

StorageIntensive

Highly Threaded

Single Threaded

SAP R3

CommercialTechnical


One Size Does Not Fit AllDatabase, file systemSAN connectedBlock storageData servicesThread level parallelismVertical and horizontal scaleBig I/O bandwidthBig memoryHighly statefulHighly reliable nodesClustering/RACData replicationGeo distributionResources to workloadPartitioningDynamic resource mgt.Cautious change mgt.In depth monitoringActive management

Web serverData from the networkSmall local storage Thread level parallelismHorizontal scaleBig network bandwidthMedium memoryStatelessHorizontal replicationDurable nodes Workload to resourcesLoad balancerPacket allocationSystem image and app.ProvisioningHealth monitoring

MPI libraries, Queuing, Job CntlData from the networkSmall local storage Instruction level parallelismFP and integer intensiveEnough memoryHorizontal scalingStatelessHorizontal replicationDurable nodes Workload to resourcesGrid softwareJob allocationSystem image and app.ProvisioningHealth monitoring

Key middlewareData flow, dataData persistence

ApplicationThroughput ServiceAvailability

VirtualizationUtilization

ProvisioningManagement

Data Web ComputeSystem Attribute


Application Utility Defined?• A platform that enables Sun and ISV partners to

offer application subscription/utility services from the grid.

• The Sun Grid platform provides dynamic provisioning, observability and lifecycle management of services> Provides a mechanism for federated, policy-based

provisioning across the grid> Captures data used for

> Measurement of resource utilization> Service quality management

> Provides a mechanism for development and deployment of systemically-distributed/peer-oriented systems

> Enabled through Sun Connection telemetry


The New Application Paradigm?

Rearden Commerce has created a fabric for the hosting of web services that enables various services to be integrated in order to complete a unit of work. They have organized these services by vertical market, and provide a common management layer to ensure security and availability. This is the paradigm that we want to ultimately both provide for, and in the case of the ecosystem, fully enable.


Application Lifecycle Challenges• Component

> Tools> Frameworks> Patterns> Partners

• Test Harnesses• Submission• Validation• Placed in Library• Change Management

• System> Tools> Frameworks> Patterns> Partners

• Integration• Verification• Change Control• Submission• Provisioning• Execution


Phase 1 Hosted Development Environment:Technical Approach

Phase 1 has minimal impact on Sun Grid infrastructure engineering as majority of development done against Servlet Facade.


Developer Environment (Phase 2)


Integrated Architectural Vision (Future)


The New Ecology


Conclusion• Grid Market has attracted a lot of attention both

academically and the industry• Still there are still of LOT of work to do. Many

issues and questions are not answered yet.• We need not just computer scientists but a whole of

others such as economists, mathematician, domain expert.


Invitation of participate• http://apstc.sun.com.sg• http://java.net• Pls send email to me if you are interesting to

work/collaborate with us> [email protected]

http://apstc.sun.com.sg/

http://java.net/

Dr Simon [email protected]@ntu.edu.sg

Thank you

mailto:[email protected]

grid market - scut market dr simon c.w. see director grid and high performance computing technology...

Documents