blackstor - world's fastest & most reliable cloud native software defined storage

BlackStor(Based on DRBD)

World's fastest & most reliable Cloud Native Software Defined Storage

How it came into the world, what’s available now and where it’s headed

1

Copyright © 2016 PROZETA & LINBIT

We were simply unhappy with SDS solutions that are available on the market these days… including Ceph.

22

Copyright © 2016 PROZETA & LINBIT 33

Is it even possible to build a new SDS standard?

4

Who we are


⬢ Since 1991⬢ Group annual turnover: 20M EUR (2015)

⬢ Two divisions➥ Hardware: Value Added Distributor & Reseller➥ Infrastructure services:

IaaS, SaaS, PaaSCE data center infrastructureIn-house development

About PROZETA

5


About PROZETA

6


About LINBIT

7

•Open Source DRBD supported by proprietary LINBIT products / services

•Hundreds of thousands of DRBD downloads

•OpenStack comes with the DRBD Cinder driver

•100% founder owned

•Offices in Europe and US

Best performing SDS block device using common off the shelf hardware

Partners

References

Only replication technology excelling at both synchronous short distance and asynchronous long distance


LINBIT’s team is made up of International Open Source experts and has successfully expanded

globally with offices in Europe and the US and partnerships across 5 continents

• LINBIT founded 2001

• European HQ; Vienna (Austria). US HQ; Portland

• Founders are High Availability pioneers

• International OSS, kernel and HA lecturers

• Most recognized Linux HA team worldwide

Incorporation of LINBIT HA

2001

2000 2003

2010

2008 2015

Release of DRBDOpen Source

Lars Ellenberg joins team as Head of Programming

2007

Release of DRBD Proxy

US Office

DRBD inLinux Kernel

DRBD in thousands of clusters worldwide

2016+

DRBD9release

9

DRBD in OpenStack

PROXY

About LINBIT


PROZETA & LINBIT together

9

PROZETAcloud experts

LINBITstorage & HA gurus

COMBINED KNOW-HOW

BEST POSSIBLE STARTING POINT FOR A NEW SDS

10

So what do we really need?


SDS Requirements

We wanted SDS on the level of Enterprise storage BUT based on a cost-effective open-source technology we understand.

11

⬢ High Stability⬢ High Reliability⬢ High and Predictable Performance⬢ Scaling options - both Horizontal & Vertical⬢ Modest resource use⬢ Simple to manage (plus repair in case of disaster)⬢ Possible to get info for deep monitoring & metrics⬢ Extensible: features like Compression, Deduplication, etc.

12

What’s needed for serious cloud storage? What’s available on the market?


What Are The Options?

13

⬢ IaaS = Volumes / Block Storage

⬢ Servers + Disks + Software

⬢ Local storage

➥ not scalable

⬢ iSCSI

➥ software defined &

redundant: hard to find

⬢ GlusterFS, LizardFS, …

➥ too complex to operate...

Goal:

“Next year I’d like to try out a different storage software solution…

...and I can!”


Situation on the Storage market

14

Open-source:

⬢ Often re-inventing wheel⬢ Too old solutions⬢ Too complex solutions⬢ Over-engineered⬢ Often unstable

Proprietary:

⬢ Often very expensive⬢ Closed source⬢ Not future proof (Software-defined)⬢ What if something goes wrong?


Look at Ceph...

15

● No compression

● No deduplication

● Ineffective thin provisioning

● Extremely CPU hungry

● Bad write performance / high latency - SLOOOOOOOOOW…

High TCO due to the need for extra hardware, unpredictable performance

Perfect integration with OpenStack

VERY stable BUT very difficult to fix potential issues in a timely manner

15


Need Software Defined Everything...

16


Need True Cloud Storage...

17


...that is Fast, Reliable, Scalable

18


Definitely requires a solid underlying base technology …

Something new, right?

19


And then we (re-) discovered

DRBD!

20

21

What is DRBD?


What is DRBD?

22

⬢ 15+ years on the market⬢ Developed by a stable company with great reputation⬢ More than 250.000 production installations worldwide⬢ Kernel modules (DRBD core) are standard part of Linux kernel⬢ Super-reliable and super-fast!


DRBD in 3 minutes

23


Why is it so freakin' fast?

24

⬢ Native Linux block device - not a block device emulation on top of an object store

⬢ Highly optimized meta data layout

⬢ The SDS solution leverages on proven technology

➥ LVM or ZFS’ zVols

⬢ Very well suited for hyperconverged compute and storage


DRBD - Key Features

25

⬢ automatic resync after node or connectivity failure➥ direction, amount, no full resync needed

⬢ performs under a Linux kernel implementation➥ 160k IOPs measured (on SSDs, of course)

⬢ multiple volumes per resource (replication group)➥ write order fidelity within resource

⬢ comes with Pacemaker integration⬢ synchronous and async replication (LAN and WAN)⬢ In Linux upstream since 2.6.33 (released 2010)


DRBD - New Features in 9.x

26

⬢ Up to 32 nodes per resource

➥ Fixes the drawbacks of stacking

⬢ Auto promote

⬢ Transport abstraction (TCP, SCTP, RDMA)

⬢ DRBD Manage


DRBD Manage

27

⬢ Provisioning solution for DRBD⬢ Integrated with OpenStack, OpenNebula, Docker, Proxmox, …⬢ Volumes⬢ Snapshots⬢ Replica count⬢ Unlimited number of nodes⬢ Distributed

DRBD Kernel driver

drbdsetup

drbdadm

drbdmanage daemon

device mapper

dmsetup

LVM tools

drbd cinder driver Drbdmanage CLI tool


DRBD Roadmap

28

29

Let's rock!


First implementation

30

⬢ DRBD Kernel modules

+ DRBD Utils

+ DRBD Manage

⬢ DRBD Cinder driver⬢ Monitoring & Metrics⬢ Backup


Stability & Reliability

We have tried to totally break it a million times...

with NO SUCCESS!

Just what you would expect in a production environment

31

32

So what now?

BlackStorWorld's fastest & most reliable

Cloud Native Software Defined Storage

33


BlackStor

34

JSON HTTP CLI WEB UI

API

CONTROL ENGINE MONITORING ENGINE

MESSAGE BROKER

AGENT

DRBD

AGENT

DRBD

AGENT

DRBD

AGENT

DRBD

BI-DIRECTIONAL INTEGRATIONS

TASK MANAGEMENT

SCALING

DISASTER RECOVERY

STATE & PERFORMANCE

METRICS

ANALYTICS

35

What more to add?


Policy-driven storage

36

⬢ Use simple policies to control:➥ Replication strategy➥ Data placement➥ Balance speed & data protection (RPO)➥ QoS & SLA➥ Backup➥ … and various other knobs

⬢ Assign policies to / by➥ Storage objects➥ Consumer objects

(OpenStack tenant, Cinder volume metadata, ...)


Multi-tenant & Multi-cloud

37

⬢ Use the same storage pool for➥ Multiple OpenStack tenants➥ Multiple OpenStack clusters➥ OpenStack & Docker➥ …..

⬢ We have QoS, so no issues with multi-tenant environment

⬢ Linked authentication➥ Use Keystone for storage authentication➥ Allows OpenStack users to manage resources (if allowed)


Scaling

38

⬢ Horizontally: Scale-out➥ Add nodes to grow cluster➥ Hyper-converged nodes (more nodes, lower capacity per node)

⬢ Vertically: Scale-up➥ Low CPU usage➥ High density storage (high capacity, high iops)➥ 76 TB SSD & 1-2M IOPs in one node? Why not?

⬢ Single volume striped across multiple nodes w/ read balancing➥ Extra boost in performance


QoS

39

⬢ Limit IOPs per➥ Volume➥ Volume group➥ Consumer object (OpenStack tenant, …)➥ Consumer (OpenStack cluster for multi-cluster environment)

⬢ Set priority➥ Set priority per policy

⬢ Alpha version - many limitations but usable⬢ Bandwidth limitation not yet implemented


Backup & “Recycle Bin”

40

⬢ Ability to roll-back without any actual recovery process➥ Storage snapshots (volume roll-back)➥ Deferred delete (volume undelete)

⬢ Backup to another host

➥ Running in background➥ Simple policies➥ No performance degradation➥ Encrypted over the wire

⬢ Recover anything➥ Single volume➥ Full storage


Fully Cloud Aware Storage

41

Everyone's talking about it, but there is none as of now! What it means?

● Bi-directional interfaces (OpenStack, Docker, etc…)● BlackStor keeps all informations about your volumes & VMs inside● Simply define QoS/SLA for OpenStack tenant or VM ….

…. and see if SLA in not breached in the Web UI


Deep insight

42

⬢ Cloud Native - Data + all metadata in one place

⬢ CLI, Web UI⬢ Real-time status

⬢ Metrics➥ IOPs, bandwidth➥ Latency distribution➥ Storage health

(software, hardware)➥ … and many more


BlackStor - Roadmap

43

⬢ Short-term➥ First BlackStor release (incl. new Cinder driver)➥ Multi-tenant, Multi-pool, ACLs, Policy driven➥ Web-based interface, single API, CLI tools➥ Backup, Scale-up & Scale-out➥ Simple installation (DEB/RPM packages, Puppet/Salt recipes)

⬢ Mid/Longer-term➥ Advanced features (QoS)➥ Encryption (both over the wire & data at rest)➥ New drivers (Nova, Docker, Kubernetes, Windows ?)➥ Backup policies

44

How does it work in the real world?(It works in theory… does it work in practice?)


Performance - DRBD by itself

45

● HW○ 2x IBM 8247-22L○ Power 8 2 sockets○ 128 GB RAM○ Mellanox 100GBps InfiniBand○ HGST Ultrastart NVMe SSDs

● Ubuntu Xenial on bare metal● DRBD 9.0.1 & RDMA Transport 2.0● fio 2.2.10● Random IO


Performance - DRBD itselfWrite 2 node DRBD

46

Write Baseline


Performance - DRBD itselfRead through DRBD

47

Read baseline


Performance - DRBD itselfDRBD with read balancing

48

Read baseline


DRBD vs CEPH

49

⬢ 5-node clusters➥ 2 Class-D SSDs each➥ 2x E5-2650-L CPUs each➥ 128GB RAM each➥ 10GbE each

⬢ Attached VM - 8-core CPU,16GB RAM

⬢ Ubuntu 14.04.5, Kernel 3.16⬢ DRBD9 vs Ceph 10.2.3⬢ fio 2.14


DRBD vs CEPH

50

Sequential write (IOPS) CPU usage


DRBD vs CEPH

51

Random write (IOPS) CPU usage


DRBD vs CEPH

52

Sequential read (IOPS) CPU usage


DRBD vs CEPH

53

Random read (IOPS) CPU usage

54

Okay, I’ll launch this into production…but who’ll integrate it? what if something happens?


Support?

55

Yes - end-to-end!

Including professional services

56

Let's sum it up!


Building a new cloud-native SDS is not an easy task...

57


...but we went for it anyway.

Why? Because...

➥ nothing reliable out there

➥ great performance & production ready

➥ sweet spot between open source CEPH &

expensive Enterprise storage58


What do you get with this solution?⬢ Production ready⬢ Stable & Reliable⬢ Great & Predictable Performance⬢ Simple to implement & manage⬢ Includes deep metrics information⬢ With additional support options

...and last but not least….

⬢ Fantastic TCO - it doesn't cost the Earth!


Q&A6060


Thank you!

PRO-ZETA a.s.

Prague, Czech Republic

[email protected]

www.prozeta.eu

www.tier5.cloud

61

LINBIT HA Solutions GmbH

Vienna, Austria

[email protected]

www.linbit.com

61

Copyright©2016PROZETA&LINBITThankyou!PRO-ZETAa.s.Prague,CzechRepublicprozeta@prozeta.euwww.prozeta.euwww.tier5.cloud58LINBITHASolutionsGmbHVienna,[email protected]

Klikněteapřidejtepoznámky

Zobrazení prezentace ve formátu HTMLMožnosti obrázku Přebarvení

Světlé 2

Nastavení

Průhlednost

40%

Jas

Kontrast

Resetovat úpravy

mailto:[email protected]


http://www.prozeta.eu

http://www.prozeta.eu

http://www.tier5.cloud

http://www.tier5.cloud



http://www.linbit.com

http://www.linbit.com

https://docs.google.com/a/prozeta.eu/presentation/d/1ECDepAsJebXGHnO8Vr5ftKy_vzsTuB_Xb7zQW-8pOU8/htmlpresent

https://docs.google.com/a/prozeta.eu/presentation/d/1ECDepAsJebXGHnO8Vr5ftKy_vzsTuB_Xb7zQW-8pOU8/htmlpresent

blackstor - world's fastest & most reliable cloud native software defined storage

Internet