Upload File

dynamic resource management for virtualization hpc environments xiaohui wei college of computer...

  • Home
  • Documents
  • Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA
15
Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on October 17-20 2011-10-19

Upload: gavin-oleary

Post on 27-Mar-2015

218 views

Category:

Documents


5 download

Report

Tags:

TRANSCRIPT

Page 1: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

Dynamic Resource Management for Virtualization HPC Environments

Xiaohui Wei

College of Computer Science and Technology

Jilin University, China.

1PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on

October 17-20

2011-10-19

Page 2: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

2PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on

October 17-20

2011-10-19

Page 3: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

3PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on

October 17-20

2011-10-19

Page 4: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

4

Introduction

• Virtualization technology– “Most application and system software will be running

on Virtual Machines (VMs) instead of physical machines

in the near future.” ----Technical Report of UC Berkeley

on cloud computing, 2009

• Different levels

– OS-level virtualization• Virtual Machine Monitor (VMM): Xen, VMWare, Virtual Box

– High level component virtualization (Virtual

infrastructure)• Virtual network, Virtual cluster, Virtual resource manager.

PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on October 17-20

2011-10-19

Page 5: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

Virtualization in Clouds

Resource scheduling (Virtual resource and traditional resource)

5PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on October 17-

20

2011-10-19

Page 6: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

6

Summary of Our Works

• Virtual infrastructure management

– LIve Migration-Enabled Virtual Infrastructure (LimeVI)• Virtual network

• Virtual cluster live migration

• Concurrent migration protocol

• Cloud management

– Elastic Virtual Cluster (EVC)• Support per-job Virtual Cluster

• Integrated with CSF

• Parallel Job Scheduling

PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on October 17-20

2011-10-19

Page 7: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

7PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on

October 17-20

2011-10-19

Page 8: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

8

Dynamic Virtual Cluster

• Dynamic Virtual Cluster management

– Dynamic in the construction phase

(customization)• Network topology, scale, OS type, cluster software, application software.

– Expansion and shrinkage in scale

Existing researches focus on dynamic VC management in the construction phase,

little work has been done to ensure reconfiguration of VC topology in runtime.

– Virtual cluster live migration

• Under-developed, lack of concurrent support.

PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on October 17-20

2011-10-19

Page 9: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

9

Parallel Job Migration

• Runtime status migration

– Memory status migration • Process migration (Condor checkpoint library)

• Virtual machine live migration (VMware-VMotion, XEN-live

migration)

– File system status migration (image, WAN)

• Communication status migration – Stop-and-wait model (MPICH-G-DM, CoCheck, MPI-

Mitten)

– Message-logging model (MPICH-GF, MPICH-V)

PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on October 17-20

2011-10-19

Page 10: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

10

Objectives and Innovation

• Existing problem of related works

– Communication status inconsistency;

– Single VM live migration prolong virtual cluster

migration.

• Objectives

– Fexible virtual infrastructure (LimeVI)

– Concurrent LIve Migration Protocol (CLIMP)• Preserve communication status consistency (message

buffering)

• Concurrent Virtual Cluster live migration

PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on October 17-20

2011-10-19

Page 11: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

11

Live Migration-Enabled Virtual Infrastructure

(LimeVI)• Flexible WAN Virtual Infrastructure (VI)

PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on October 17-20

LimeVIVirtual Network

Physical networkWAN

VC2VC1

C

D E

1 2 6 4 8 3 7 5

573

881 26 4

A B LAN2

LAN1

LAN3

Daemon

Host

VM

• Daemon • package filtering, virtual routing, tunneling;

• package buffering ,Migration protocolVM8 on hostC is migrating to hostB to peruse better integer network performance.

2011-10-19

Page 12: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

12

LimeVI Daemon

PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on October 17-20

• Buffer Chain (BC): store packages for migrating VMs– Every Daemon maintains one buffer chain for each migrating VM;

– Related VMs;

– Only logically belongs to local Daemon.

NIC of HostC

Releasing Arbiter

Capturing Arbiter

Virtual Routing Table

IP tunneling

BC for VM1

BC for VM8

VM1 VM2

i+n

i

VM6VM4

Buffer chains

Local VMsVM7 VM8

2011-10-19

Page 13: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

13

CLIMP: Concurrent Live Migration Protocol

• Solutions

– Coordinated distributed Daemons

– Logical BCs among related LimeVI Daemons

– Complete block and bufferring during migration

PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on October 17-20

2011-10-19

Page 14: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

1414

Elastic Virtual Cluster

• LimeVI : virtualized network and VC live

migration

• CSF – meta-scheduler : resource

allocation/reallocation for virtual clusters

(parallel jobs)

• CSF : scheduling plugin

PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on October 17-

20

2011-10-19

Page 15: Dynamic Resource Management for Virtualization HPC Environments Xiaohui Wei College of Computer Science and Technology Jilin University, China. 1 PRAGMA

15

THANK YOU!

PRAGMA 21 Workshop, Sapporo, Hokkaido, Japan on October 17-20

2011-10-19


Folleto Grupo Pragma

Pragma FortressSSH 5.0

gnu.huihoo.org · ii 2.38 Pragma Contract Cases:::::::::::::::::::::::::::::::::::::: 22 2.39 Pragma Convention Identifier:::::::::::::::::::::::::::::::: 23 2.40 Pragma CPP Class

Parallel Programming Course OpenMP - Intel · #pragma omp parallel {#pragma omp task foo(); #pragma omp barrier #pragma omp single {#pragma omp task bar();}} task : Multiple foo()

OpenMP in Actiongabriel/courses/cosc4397_s10/ParCo_16_OpenMP.pdf · #pragma omp parallel #pragma omp sections {#pragma omp section X_calculation(); #pragma omp section y_calculation();

PRAGMA AUTONOMOUS TRANSACTION

GNAT Reference Manual - GNU Projectgcc.gnu.org/onlinedocs/gnat_rm.pdf · iv 2.134 Pragma Postcondition ::::: 68 2.135 Pragma Post Class ::::: 70 2.136 Pragma Rename Pragma ::::: 70

GNAT Reference Manual - GNU Projectgcc.gnu.org/onlinedocs/gcc-7.2.0/gnat_rm.pdfiv 2.134 Pragma PrefixException Messages ::::: 67 2.135 Pragma Pre Class ::::: 68 2.136 Pragma Priority

THE PRAGMA-IDEOLOGICAL IMPLICATIONS OF USING REPORTED ... · THE PRAGMA-IDEOLOGICAL IMPLICATIONS OF USING ... i.e. absolving the journalist/the news ... The pragma-ideological implications

Mini Pragma - Filkab.com

GNAT Reference Manual - GNU Projectiv 2.134 Pragma PrefixException Messages ::::: 67 2.135 Pragma Pre Class ::::: 68 2.136 Pragma Priority SpecificDispatching ::::: 68 2.137 Pragma

Pragma Corporate Finance · Pragma Corporate Finance . PRAGMA Table of Contents 2 Page Section 1 Pragma Overview 3 Section 2 Recent Transactions Completed by Pragma 11

Historia Pragma Consultores

1 Dr. Xiaohui Wei College of Computer Science and Technology, Jilin University, China CSF4 Tutorial The 3rd PRAGMA Institute, Penang Malaysia, 2008-10-21

Company Profile Pragma Africa - Pragma Latam · The Pragma Way 4 The Pragma Group Pragma Holdings acts as an investment vehicle for the group and provides strategic and operational

Pragma 07 low

A Pragma-Rhetoric Analysis of Persuasion - …euacademic.org/UploadArticle/2258.pdf · A Pragma-Rhetoric Analysis of Persuasion ... This research is a pragma-rhetorical analysis of

Medi Pragma 2012

Review Article Effect of migraine on expression of PACAP ... · Department of Human Anatomy, Jilin Medical University, No. 5 Jilin Street, Jilin 132013, Jilin Province, China Received

Manual identidad PRAGMA

Pragma Corporate Finance Pragma Presentation_Eng.pdf · Pragma Corporate Finance . PRAGMA Table of Contents 2 ... pulp and paper chemicals, water treatment chemicals, cement grinding

«OpenMP Programming Model - Tasks» Panos Hadjidoukas · • #pragma omp parallel • #pragma omp sections • #pragma omp parallel for • “parallel for” is great for for-loops,

GNAT Reference Manual · iii 2.86 Pragma Inline Always ::::: 44 2.87 Pragma Inline Generic::::: 44 2.88 Pragma Interface ::::: 44

Pragma Catafgrlog Technfgdfgical

PRAGMA Institute on Implementation @ PRAGMA 19 Wilfred W. Li, Ph.D., UCSD, USA Xiaohui Wei, Ph.D., JLU, PRC Hosted by JLU Changchun, Jilin, PRC, Sept 13,

Progress from PRAGMA 7 PRAGMA 8 Workshop 3 May 2005 Singapore Bioinformatics Institute

Lixian Xi @pragma 19 @pragma 19 Changchun, Jilin,Sep 15,2010 College of Computer Science and Technology, Jilin University, ChangChun, P.R.China Lixian

Pragma overvannsrør produktkatalog

Sinosteel Jilin Carbon

SDx Pragma Reference Guide - Xilinx€¦ · pragma HLS array_map Added discussion of clock cycle expectations for horizontal and vertical array mapping. pragma HLS interface Added

Pragma On Key

GNAT Reference Manual - GNU Project · iv 2.134 Pragma Post Class ::::: 69 2.135 Pragma Rename Pragma ::::: 70 2.136 Pragma Pre ::::: 70

Pragma Report

Loop Optimizations in LLVM: The Good, The Bad, and The Ugly#pragma _CRI unroll #pragma _CRI fusion #pragma _CRI nofission #pragma _CRI blockingsize #pragma _CRI interchange #pragma

Pragma celá