rakuten's private cloud
DESCRIPTION
Explains the origin of Private Cloud at Rakuten, how we built it, promote it, and operate it.TRANSCRIPT
Rakuten’s Private Cloud
Jonathan Levine
http://www.rakuten.co.jp/
2
What’s Rakuten?
Rakuten Facts • Founded in 1997 by Hiroshi Mikitani
• Over 9,000 staff worldwide
• Rakuten Ichiba and Rakuten Travel are largest businesses;
we have a total of more than 50 services
• Domestically, more than 130 million products offered by
more than 130,000 providers
Corporate History - Domestic 1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
Corporate History – Global 1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
Domestic Marketplace Sales Growth
Source: Rakuten
*: Domestic E-Commerce = Mall (Fixed-price, Group purchase), Mobile, Business, Golf, Books, Auction (Flea market), download, tickets,
Internet Supermarket, Check Out, Off-track betting, toto, ShowTime, Media rental, Dining , Salon (added from Q2/12), and Mart and
Kenko.com (added from Q3/12)
International Marketplace Sales Growth
Source: Rakuten
*: Marketplace = Taiwan Rakuten Ichiba + Tarad Premium Mall + Lekutian + Buy.com marketplace + PriceMinister marketplace +
Rakuten Belanja Online + Rakuten Deutschland + Play.com marketplace + Rakuten Brasil marketplace (from Q2/12).
8
Why is Rakuten Ichiba Different?
Our Mission
Empowerment
10
Unique E-Commerce Model
Rakuten’s Empowerment Model is Unique and Difficult to Copy
11
Typical E-Commerce Site
12
Rakuten’s E-Commerce Site
13
Why Cloud?
My First Computer
Moore’s Law
Source: Wikipedia
17
Server CPU Utilization, by Service
新サービス系サーバー別 CPU稼働率(2009年3月)
0
10
20
30
40
50
60
70
80
90
100
wfb
tool
wpl
aza
mis
ml
wde
li
wenq
wm
oney
wdi
gi
win
cu
woubo
wis
eek
ddin
wnik
ki
btm
ail
bpla
za
wis
dir
bdig
i
wsp
ace
ddeli
wtm
ail
bfbt
ool
wap
itic
wps
p
wga
me
ww
om
an
apde
licti
wpo
rtal
gl
bnik
ki
wdi
giad
m
wm
ile
wde
liim
g
bmedi
a
asear
chr
wbr
tic
wpo
rtal
adm
mti
c
wga
meap
p
bauto
サーバー群
CPU使用率(%)
CPU使用率最大値
CPU使用率平均値
10%
Domestic Marketplace Sales Growth
Source: Rakuten
19
Cloud at Rakuten
20
Step 1: Virtualization
Virtual Server Pool
Server Server
Shared Storage
Hard Disk
Storage Network
Old: 1 physical == 1server New:1 physical == n servers
System
Benefits of Virtualization
•Faster Deployment Time for New Servers •Automatically Handle Downtime •Improve Equipment Utilization •Easy Replacement of Hardware
OS
App
OS
App
OS
App
OS
App
OS
App Virtual Server Pool
21
“Pcloud” production Environment
Server Farm:
104 Physical Storage Farm:
14 Physical
iSCSI
22
Private Cloud Progress – Stage 1
VMs
Why so Slow?
Technology.
25
“RIaaS" production Environment
Fabric Network
(10G, FCoE)
3PAR Storage
Blade Server VMware
26
Private Cloud Management Tool
・VM management
・Server Setup (with Chef)
・DNS setting
・SLB control
Not Just Technology.
Not Just Technology.
It’s an Internal Business.
• Tools
• Training
• Support
• Roadmap
• Communication
== Product Management
RIaaS is crossing over the Chasm!!!
We are here!
Early Adopter Early Majority Late Majority
Proof of solution Willing to use Step by Step
Test -> Production
Case study
Sensitive
Communication Word of mouth Interactive Passive
SLA No Risk No Return Reasonable requirement Demanding
Priority Speed
Experience
Balanced Risk/Return Stability
PCloud
RIaaS
32
Private Cloud Progress – Stage 2
Jul-1
0
Aug-1
0
Sep-1
0
Oct-
10
No
v-1
0
De
c-1
0
Jan-1
1
Feb
-11
Ma
r-1
1
Apr-
11
Ma
y-1
1
Jun-1
1
Jul-1
1
Aug-1
1
Sep-1
1
Oct-
11
No
v-1
1
De
c-1
1
Jan-1
2
Feb
-12
Ma
r-1
2
Apr-
12
Ma
y-1
2
Jun-1
2
Jul-1
2
Aug-1
2
Sep-1
2
Oct-
12
No
v-1
2
De
c-1
2
Jan-1
3
Feb
-13
Ma
r-1
3
VMs
33
Case Study
34
Exciting Event …?
35
20
05/1
/1
20
05/6
/1
20
05/1
1/1
20
06/4
/1
20
06/9
/1
20
07/2
/1
20
07/7
/1
20
07/1
2/1
20
08/5
/1
20
08/1
0/1
20
09/3
/1
20
09/8
/1
20
10/1
/1
20
10/6
/1
20
10/1
1/1
20
11/4
/1
20
11/9
/1
20
12/2
/1
20
12/7
/1
20
12/1
2/1
SuperSale Network Effect
36
Private Cloud for SuperSale
0
1000
2000
3000
4000
5000
6000
Jul-1
0
Aug-1
0
Sep-1
0
Oct-
10
No
v-1
0
De
c-1
0
Jan-1
1
Feb
-11
Ma
r-1
1
Apr-
11
Ma
y-1
1
Jun-1
1
Jul-1
1
Aug-1
1
Sep-1
1
Oct-
11
No
v-1
1
De
c-1
1
Jan-1
2
Feb
-12
Ma
r-1
2
Apr-
12
Ma
y-1
2
Jun-1
2
Jul-1
2
Aug-1
2
Sep-1
2
Oct-
12
No
v-1
2
De
c-1
2
Jan-1
3
Feb
-13
Ma
r-1
3
VMs
37
Next Steps
High Capacity Racks
PatchPatch
EqualLogic
R610
R610
21
543
876
11109
141312
17 R6101615 R610
2019 R61018
27 R610
2423 R610
25 R610
31 R61030
28
353433 R61032
41 Patch4039383736
21 R61022
26
29 R610
42 Patch
39 Nexus 2232PP
2U4N32
42 Nexus 2232PP41 Nexus 2232PP40 Nexus 2232PP
383736353433
312U4N30
292U4N28
272U4N26
252U4N24
232U4N22
212U4N20
192U4N18
172U4N16
152U4N14
132U4N12
52U4N4
112U4N10
92U4N8
32U4N2
1Management 48port switch
72U4N6
Standard Rack
(3KVA)
Hicap Rack
(15KVA)
48 Nodes
9 Cables
12 Nodes
36 Cables
Site Recovery Manager for Migration
Old DC New DC
40
Global DC Expansion
EU DC
Japan DC
U.S. DC
Private Cloud Roadmap
• IaaS (Compute as a Service) improvements
– Add APIs
– Add App team access
• DBaaS (Database as a Service)
• STaaS (Storage as a Service)
• PaaS (Platform as a Service)
42
Final Thoughts
43
•12 Employees
•27,000,000 users
•2 years
44
•31 Employees
•#16 site in US (as of May, 2012; source: Alexa)
•30 months
45
But…
• Technology is really important…
• … but it isn’t everything
• Cost is really important…
• … but it isn’t everything
• Make sure you can have enough
scale to be competitive…
• Then compete to win!
Thank you!