Success StoryYahoo Japan Adopts Clustered Data

ONTAP as a New OpenStack Infrastructure

for Service Developments. Deploying

Multiple VMs While Reducing Storage Costs

Adopting OpenStack as the cloud infrastructure

for development to support a rapid growth of

services

Yahoo Japan Corporation (hereinafter referred to

as Yahoo) is an information and communications

company operating Japan’s largest portal site

Yahoo! Japan. Yahoo! Japan, is a "problem-

solving engine" that solves people's and society's

problems with IT. It has developed a number

of user-focused services, such as search,

news, mail, auctions, and shopping. In recent

years, it has also set "detonation velocity,"

focusing on speed as the goal. They have

continued challenging themselves to develop

amazing services through a bold commitment

to "challenging 10-fold, failing 5-fold, and

succeeding 2-fold."

Since 2009, Yahoo has operated a private

cloud infrastructure, mainly used by application

developers, in order to consolidate physical

servers that increased rapidly in response to

growth. Utilizing an API that was developed

Industry

• Information and Communications Technology

The Challenge

• Build a cloud infrastructure with excellent

scalability that meets the needs for rapid

business growth

• Deploy an OpenStack infrastructure that can

respond flexibly to the diverse needs of each

service

• Adopt storage systems to support OpenStack

Block Storage

The Solution

• NetApp Clustered Data ONTAP® 8.2

- RAID-DP, FlexVol, Snapshot

• NetApp FAS system with active-active

configuration

• NetApp Flash Cache

• NetApp SnapMirror ® Software

• NetApp OnCommand™ management

software

Benefits

• Support non-stop operations of the cloud

infrastructure through high-availability storage

• Enable deployment of a large number of VMs

while maximizing disk utilization with FlexVol

• Possible development in disaster recovery

capacity between sites by utilizing NetApp

data protection feature

KEY HIGHLIGHTS

• Since 2009, Yahoo Japan Corporation has operated a private cloud infrastructure for

developing applications which support a variety of services.

• The number of vir tual machines increased to thousands along with a rapid growth

of services, causing the company to face major issues such as the scalability and

operational load of the cloud infrastructure.

• Yahoo decided to adopt open source OpenStack as a cloud infrastructure for the next

generation since it can respond flexibly to in-house feature improvements utilizing APIs.

• They adopted NetApp Clustered Data ONTAP, enabling non-stop operations of the

cloud infrastructure and supporting OpenStack Block Storage (development code

name: Cinder).

• Four nodes of NetApp FAS3220AE have been deployed, and a total of approx. 20,000 VMs

are in operation on the OpenStack infrastructure.

• Through advanced integration with FlexClone, they achieved a very high disk utilization while

deploying a large number of VMs.

• In the future, the OpenStack infrastructure will be deployed to more data centers, and

disaster recovery will be enhanced through storage integration between data centers with

SnapMirror.

ABSTRACT

in-house, the cloud infrastructure provides

an environment where resources such as

virtual machines, storage, and the network

can be controlled flexibly through a Web

browser. However, as the number of virtual

machines increased to one thousand units,

new challenges emerged: difficulty in efficient

expansion of hardware, increase in operational

management load, and so on.

The company has decided to adopt "OpenStack"

which can provide a more advanced and open

cloud infrastructure for the purpose of optimizing

data center resources and achieving more

efficient operational management. The company

has watched developments with OpenStack

since around 2010. By 2012 OpenStack had

significantly more components and reached a

level that has sufficient capacity for the company's

cloud infrastructure. Then, in the second half of

2012, the company started to consider introducing

OpenStack. In March 2013, they launched the

company's first OpenStack project.

"Although there are some solutions for commercial

server virtualization with which you can operate

a few thousands virtual machines, we decided

to deploy an open source cloud infrastructure

because we have made a variety of feature

improvements in-house for the convenience of

our users. In particular, we had the impression

that OpenStack has a very active community

and that its experts provide improvements and

bug fixes promptly. In addition, OpenStack's

rich components and APIs enable the cloud

infrastructure to be customized flexibly. For

example, OpenStack Identity (development

code name: Keystone) is used to unify account

information, which is managed separately for

each service using the account authentication

in our OpenStack infrastructure. We expect that

this will remove the need for traditional in-house

API developments and will promote efficiency for

infrastructure development." -- Daisuke Masaki

Non-stop operations and OpenStack integration

are the reasons for adopting Clustered Data

ONTAP

Possessing extensive operational experience

with Yahoo’s various service and development

platforms, NetApp products were selected

for storage systems that support OpenStack

infrastructure. Yahoo mainly used 7-Mode

for operations; however, they are currently

integrating scale-out Clustered Data ONTAP

to pursue a scalable storage infrastructure that

allows for 24/7 non-stop operations and that

can respond to the rapid growth of services. In

particular, the current Clustered Data ONTAP

8.2 supports a QoS (Quality of Service) feature

to optimize the storage performance for each

co-existing service and data synchronization

features between storage clusters (SnapMirror)

to help in creating a disaster recovery

environment. Another key development is a

NetApp storage dedicated driver (NetApp

Unified Driver for Clustered Data ONTAP),

which supports OpenStack Block Storage

(development code name: Cinder).

"When you use a dedicated driver provided

by NetApp, you can intelligently integrate

OpenStack Block Storage (hereinafter referred

to as Block Storage) processing into various

NetApp storage features. For example, when you

create a virtual machine on the Block Storage,

you can use the NetApp virtual cloning feature

(FlexClone) to deploy virtual machines. Without

FlexClone, disk space is physically consumed,

depending on the number of virtual machines.

FlexClone helps you minimize the disk usage

and lower the load on the disk drive at the same

time. NetApp also contributes significantly to the

OpenStack community, and this positive attitude

was also key to choosing NetApp products."

-- Noriyasu Atsuta

NetApp storage with 4 nodes total deployed in

data centers

Yahoo conducted a full-scale model testing

in conjunction with the vendors and system

integrators in October 2013. In the test, they

checked to see the differences between

7-Mode, commonly used in the company, and

Clustered Data ONTAP. They also checked

basic operations such as volume creation and

deletion for the Block Storage and snapshot

creation in the OpenStack environment--and

measured virtual machine performance. Based

on a number of these validation results, they

designed the overall system and established

an OpenStack infrastructure from February

to March 2014. The OpenStack infrastructure

adopted was OpenStack Havana Release, the

latest release at the time.

"We deployed Clustered Data ONTAP based NetApp storage

systems that quickly added support for integration with OpenStack

Block Storage."(Right)

Yahoo Japan Corporation

Systems Management Group

Site Operations Headquarters

Infrastructure Technology Department 3, Storage

Daisuke Masaki

(Left)

Yahoo Japan Corporation

Systems Management Group

Site Operations Headquarters

Infrastructure Technology Department 3, Storage

Noriyasu Atsuta

To support the OpenStack infrastructure, NetApp

FAS3220AE storage systems are deployed in

multiple node configurations with redundant

storage controllers. Storage controllers improve

overall storage read performance by incorporating

cache modules (NetApp Flash Cache) based on

flash technology. For access and management of

NetApp storage in the OpenStack environment,

a dedicated driver for Block Storage is used.

With this driver, you can choose iSCSI or NFS

as storage protocols, but the company has used

NFS for its convenient file-level storage access.

A virtual machine is started from its image file in

the volume at the moment, but they are planning

that it can also be started from its volume in the

future. The review phase is almost complete,

and a virtual machine is expected to be started

both from its image and from its volume soon.

"In our OpenStack infrastructure, a number

of virtual machines operate at the same time,

which produces a significant load on the storage

system. In addition, most of the virtual machines

are deployed by FlexClone, and local access

to the disk drive is likely to occur. Although our

NetApp storage is equipped with a SATA drive

due to its cost effectiveness, the system still

demonstrates high performance with Flash

Cache. In the future, access load is expected

to increase along with the diversification of

services. We will consider deploying an SSD-

based advanced caching technology, "Flash

Pool," if higher write performance is required. We

believe that we can use NetApp storage with

confidence, now and in the future, thanks to its

various options for further storage performance

enhancements." -- Daisuke Masaki

Data backup and restore for virtual machines are

performed using an OpenStack data protection

feature, but in case of an emergency, the

infrastructure operations team also has a backup

system in place that leverages NetApp storage

data protection. They are performing fine-tuned

multi-generational backup with Snapshot as well

as data synchronization to another volume with

SnapMirror. At present this is limited to the use

of SnapMirror within the same cluster, but they

plan to extend it to different clusters in the same

data center--or SnapMirror between different

clusters in the future. With this plan, they intend

to enhance the business continuity levels and

create a robust cloud environment that does

not stop services even if a large-scale disaster

would occur.

Planning to expand the OpenStack

infrastructure, assuming 50,000 virtual machines

The OpenStack infrastructures (in two locations)

are both operating with stability and have a high

degree of affinity with Clustered Data ONTAP.

These OpenStack infrastructure systems house

approximately 20,000 virtual machines in total

(as of April 2014), and their disk utilization is

maximized through advanced integration with

FlexClone. According to research by Yahoo, the

proportion of disk space that is actually used is

low compared to the sum of spaces allocated

to each virtual machine. These results show

that the company could reduce storage cost

effectively. If the OpenStack infrastructure will

be simultaneously used in many services in the

future, the storage I/O that occurs in one service

may affect other services. To address such

issues, the company plans to introduce the QoS

feature of Clustered Data ONTAP and to optimize

the storage I/O for each service.

The company also intends to expand the entire

infrastructure in stages to meet the expected

growth, assuming that 50,000 virtual machines

would start operations on the OpenStack

infrastructure by the end of 2014. In principle,

they plan to deploy OpenStack clusters with

approximately 4,000 operating virtual machines

in various data centers nationwide. Major version

upgrades for OpenStack are planned once

every six months. So the company intends to

always use the latest releases for any software

supporting the OpenStack infrastructure to

enhance the entire infrastructure. The company

is also expecting to link OpenStack infrastructure

systems between data centers to help disaster

recovery in the future.

"OpenStack has been upgraded almost every

half a year: the release was called "Icehouse"

in April 2014 and then "Juno" in October 2014.

In general, this six-month update cycle is

considered very short; however, Yahoo, with its

goal of "detonation velocity," needs to expand

OpenStack infrastructure rather quickly and

welcomes the speed of OpenStack upgrades.

We are carrying out technical validations of new

Approx. 20,000VMs total are in operation

Current production environment Future plan

Control

Provide Cinder Volume

Provide Cinder Volume

Approx. 20,000VMs total are in operation

SnapMirror synchronization between data centers

Control

www.netapp.com

COMPANY PROFILEOpenStack infrastructure in parallel with the

existing infrastructure operations so that we can

always deploy the latest version when building

an OpenStack infrastructure. In such cases,

we hope to integrate new NetApp storage

features and build a more attractive OpenStack

infrastructure." -- Noriyasu Atsuta

"As storage-related components, OpenStack

has Block Storage in addition to Object Storage

(development code name: Swift). Object

Storage is used to store unstructured files such

as images and videos attached to various

services as well as archiving data. We would

like to investigate how we can leverage Object

Storage in our services. A solution provided by

NetApp uses the NetApp E-Series as the Object

Storage data storage destination, and one of its

key benefits is the simple system configuration

compared with the existing storage combination.

If we decide to deploy Object Storage in the

future, we will definitely consider adopting

NetApp E-Series." --Daisuke Masaki

"Assuming virtual machines will increase up to 50,000 by the end of 2014,

we build OpenStack infrastructure in stages for our data centers"

NetApp, Inc.

NetApp provides effective storage and data management solutions

that enable dramatic cost reduction and enhance your company's

competitiveness. Discover our passion for helping companies around

the world go further, faster at www.netapp.com.

495 East Java Drive, Sunnyvale, CA 94089 USA TEL: +1 408 822 6000 URL: http://www.netapp.com E-mail: info@netapp.com

© 2014 NetApp, Inc. All rights reserved.

All specifications are subject to change without notice.

No portions of this document may be reproduced without prior written

consent of NetApp, Inc. NetApp, NetApp logo, Go further, faster,

Data ONTAP, Snapshot, FlexVol, MultiStore, FilerView are registered

trademarks of NetApp,Inc. in the United States and/or other countries.

All other brands or product names are trademarks or registered

trademarks of their respective companies. CSS-6712-0414

Yahoo Japan Corporation

http://www.yahoo.co.jp/

Headquarters Midtown Tower 9-7-1 Akasaka,

Minato-ku, Tokyo

Founded January 1996

Capital 8 billion 271 million yen

(as of March 31, 2014)

Number of Employees

Approximately 4,607

(as of March 31, 2014)

Businesses Internet Advertising business,

e -Commerce business,

Member Service business, etc.