vm visitation and resource micropayments as practical components towards cloud federations

.

Visitation Platform + Micropayments.Problem is.........that cloud federations are rare ... because no one knows how to build them

• most federations have rigid structure (can't grow)

• heterogeneous environments are a major problem• there are no examples of resource economies in federations

.Visitation 01 ......... is when VMs can visit with any cloud environment

.Micropayments 02 .....

....is a technology for resource accounting in federations01 myself+0 "A Cloud Visitation Platform for Federated Services at Network Edge" CISSE (2014)

02 myself+0 "Coins in Cloud Drives Can Use OAuth for Micropayments and Resource Metering Alike" CFI (2014)

M.Zhanikeev -- [email protected] -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 2/21...

2/21

.

The 3 Know-Hows

1. the visitation platform itself, compatible with Xen platforms 01

◦ migration, APIs, interfaces, automation

2. binary diff technology for efficient migration within a federation 03

3. cloud coin for resource metering in an open federation 02

01 myself+0 "A Cloud Visitation Platform for Federated Services at Network Edge" CISSE (2014)

03 myself+0 "「Cloudy会」DIFF法：テキスト・ファイルシステム・バイナリ差分" http://goo.gl/v8kLpQ (2015)



3/21

http://goo.gl/v8kLpQ

.

Visitation Platform Basics

Storage Sensors

…

Cloud Visitation Platform

Container


App App App …

APIs

…

Multitenancy

Physical Device

… Interfaces Billing,

Fairness

Generics, Resilience

Application Specifics

• today: VMs are running inblackboxes, zeroawareness of local hardware

• visitation: the oppositeapproach -- VMs/containersare fully aware of localresources 01

• implemented as APIs/Interfaces

• not hard -- just yum installand add your software ontop of XCP

01 myself+0 "A Cloud Visitation Platform for Federated Services at Network Edge" CISSE (2014)


4/21

.

Cloud Coin BasicsCRITICAL-ness

Distributed Resource APIs

• micropayment: payment ofa very small amount (sub-yen)

• problem: a fixedtransaction cost per pay

• solution: open coin,service can accumulatepayments (cost per batch)

• cloud coin: an ecoin storedin a cloud drive 02



5/21

.

Cloud Coin = OAuth-shared JSON file• can be a simpleJSON file stored in a cloud drive (Dropbox, ...) 02

• client - service provider: check/deduct the amount, store/changethe certificate (md5)

• service - coin provider: payments are cleared based on certificates◦ note: there is a probability of collision, the coin is lossy, loss ratio is controlled byservice statistically

Client ID

Coin Structure (JSON)

Coin ID

Coin Amount

Latest Certificate

Other Info



6/21

.

From Talk to Action


7/21

.

Talk → Action : A Practical Visitation

• say, I have (1) Hadoop and (2) a BigData replay platform 05

◦ bigdata replay on multicore, data streaming sketches, etc.

• I want to let other people/clouds to schedule jobs on my platform• I can let VMs visit with my platform to:

1. run jobs locally2. process data and run more jobs3. ...

• I need cloud coin as a quota meter◦ each VM has to let me know where its coin is before it visits

05 myself+0 "Replayable BigData for Multicore Processing and Statistically Rigid Sketching" IC2014 (2014)


8/21

.

2 Demos: BigData Replay, Cloud Coin


9/21

p4notewifi

Freehand

.

Step 1: VM Migration Automation

• the problem is to receive and run VM image as.xva in any network environment

• PV_args is the easiest way, visitationtools in VM parse network environment in /proc/cmdline1. xe... vm-import filename=?.xva2. xe ... PV_args3. xe ... vm-start

• when done, check if VM is on the network andnotify the remote client


10/21

.

Step 2: .xva diffs

• idea: keep a library ofreference .xva filesand send only diffs formigration

• 90-95% reduction involume

• diff is definitely underratedand underused in cloudstoday 03

03 myself+0 "「Cloudy会」DIFF法：テキスト・ファイルシステム・バイナリ差分" http://goo.gl/v8kLpQ (2015)


11/21

http://goo.gl/v8kLpQ

p4notewifi

Freehand

.

Step 3: the DiffHub technology

Cloud Drive

User A

User B

New

Old

Old

Diff Diff

Old version New version Binary diff

DiffHub

• work in progress

• a DiffHub for federations

• a file synctechnology where onlydiffs are transferred

• biggest merit: masterstorage is for diffs only


12/21

p4notewifi

Freehand

.

Step 3b: Docker wants diffs?

• Docker's main feature is filesystem diffs◦ .. which is a redundancy avoidance technology

• binary diffs can be more efficient than Docker filesystem layers


13/21

p4notewifi

Freehand

.

Practical Visitation: Hadoop

• problem: very small publicIP space (security, rules, etc.)

• solution: run the VMs inprivate space with Internetaccess

• demerit: outside cannotpush into your VM, but yourVM can pull anything


14/21

.

Practical Visitation: Hadoop (2)

• VM installs visitation tools,which know where to find theAPI for its PM

• the API is generic --provides any functionality

• storage, sensors, indexing,hadoop jobs, etc.


15/21

.

Public vs Private Cloud (looks like this)

• you need at least 2 PMs, 1 public + 1 private• note: hide your PM! (non-public IP on thesame subnet)


16/21

p4notewifi

Freehand

.

Practical Visitation : Cloud Coins


17/21

.

Push vs Pull


18/21

.

Visitation Platform: PULL Type


App App App …

Pull-Based Installation (of Apps)

End User

Device Provider 3rd Party

Service

Local services

One Cloud

• PULL: heterogeneous,large-scale cloudfederations, fog computing

• active probing network, sensorcloud, vehicular cloud, etc.


19/21

.

Visitation Platform: PUSH Type


App App App …

Cloud/device Owner

3rd Party Service Federated

Clouds

Local services

Federated Cloud

Manager • PUSH: trusted cloudfederation betweenlarge providers

• DC-DC cross-providerclouds


20/21

.

That’s all, thank you ...


21/21

vm visitation and resource micropayments as practical components towards cloud federations

Technology

cloud federations

cloud drives

resource micropayments

cloud environment

batch cloud coin

vm visitation

cloud drive dropbox

resource accounting