day 2 general session presentations redisconf

Salvatore SanfilippoCreator of Redis

Yiftach ShoolmanCo-Founder & CTO Redis Labs

2 Hot Topics+

1 New Trend

Redis on New Memory Technologies

DRAM NV-DIMM/PM NVMe SSD SATA SSD$0

$1

$2

$3

$4

$5

$6

$7

$8

$9

$10

0.01

0.12

1.20

12.00

120.00

1,200.00

0.10

250.00

0.10

500.001 GB cost vs. Read/Write Performance

1 GB cost Read latency Write latency

$/G

B

mic

rose

c

$9

$0.4

High DRAM Cost and Low Flash Performance

High DRAM Cost and Low Flash Performance

DRAM NV-DIMM/PM NVMe SSD SATA SSD$0

$1

$2

$3

$4

$5

$6

$7

$8

$9

$10

0.01

0.12

1.20

12.00

120.00

1,200.00

0.10

90.00250.00

0.10

25.00

500.001 GB cost vs. Read/Write Performance

1 GB cost Read latency Write latency

$/G

B

mic

rose

c

$9

$0.4$1

High DRAM Cost and Low Flash Performance

DRAM NV-DIMM/PM NVMe SSD SATA SSD$0

$1

$2

$3

$4

$5

$6

$7

$8

$9

$10

0.01

0.12

1.20

12.00

120.00

1,200.00

0.10

1.00

90.00

250.00

0.10

1.00

25.00

500.001 GB cost vs. Read/Write Performance

1 GB cost Read latency Write latency

$/GB

mic

rose

c

$9

$0.4$1

$2

Bring Redis Performance on Persistent Memories

Close to Redis Performance on DRAM

The Challenge

Before dealing with this challenge…

Is Persistent Memory Really Persistent ?

NVMe

NV-DIMM

Cloud Instance

NVMe

NV-DIMM

Failed Instance

Is Persistent Memory Really Persistent ?

Is Persistent Memory Really Persistent ?

NVMe

NV-DIMM

Failed Instance

NVMe

NV-DIMM

New Empty Instance

Data Loss

Is Persistent Memory Really Persistent ?

NVMe

NV-DIMM

Failed Instance

NVMe

NV-DIMM

New Empty Instance

Alternative Solution

Redundant Flash Array

Cloud Instance

Alternative Solution

New non-empty Instance

Redundant Flash Array

Failed Instance

Another Alternative

• Requires ultra high-speed/low-

latency network like Fiber-Channel

or Infiniband

• Latency is still at ~ 100microsec

• Expensive, complex and not

supported by cloud vendors New non-empty Instance

Redundant Flash Array

Failed Instance

Persistent Memory = Ephemeral Memory

Persistent Storage

AOF, SnapshotNVMe

NV-DIMM

Cloud Instance

Redis on Flash (RoF)

• Flash used as a RAM extender (not as a persistent storage)

• In RAM: dict & keys & hot values

• On Flash: cold values

• Values are either in RAM or on Flash

• Multi-shard; multi-threaded; asynchronous

• Pluggable storage engine (RocksDB and IBM’s ark)

• 100% compatible with the OSS Redis

Flexible RAM/Flash Ratios

Behind the Scene

Increase RAM ratio Scale-out Cluster

2500 GB2500 GB

50%50%

RoF is Fast

Avg: 2.04M ops/secMax: 2.14M ops/sec

Avg: 0.91msecMax: 0.98 msec% below 1msec:100%

Avg: 313RMB / 9.4WMBMax: 1.71RGB / 96WMB

Avg: 1.45Gbps (Tx) / 0.97Gbps (Rx)Max: 1.6Gbps (Tx) / 1.2Gbps (Rx)

>2M Ops/sec, > 1GB disk bandwidth, @ <1 msec latency

readwrite

readwrite

Master-Master Replication

Customer Perspective

• Keep the speed of Redis

• Maintain all Redis commands - simple and complex

• Flexible configuration across regions

• Any # of regions, single/multi-az/rack, any combination of nodes/shards

• Fully-managed service

How it Should Work? Start

Synced

Not synced

How it Should Work? Write

Locally

Synced

Not synced

App

App

How it Should Work? Distribute P2P

Asynchronously

Synced

Not synced

App

App

PartiallyConverged

How it Should Work?

Synced

Not synced

App

App

App

App

FullyConverged

How it Should Work?

Synced

Not synced

Our Approach to Master-Master

• Behaves like local Redis, performs like local Redis

• Based on CRDT (Conflict-free Replication Data Type)

• Supports Strong Eventual Consistency• No consensus protocol; no need for R + W > N

• Any two nodes that have received the same (unordered) set of updates will be

in the same state

• More to come…

1 New Trend

Foreword

If you are here …

You’re probably a geek

Or a geekette (the female geek)

• Dealing with simple problems

• Writing code in shitty languages

• Using silly npm packages

• Not refactoring your poor old architecture because of politics

You are tired of…

And you really want to solve real problems

So what are your options ?

Option #1 – Create your own startup

50% of startups fail in the first year

and 

95% percent fail within five years

Option #2 – Create a Redis Module

Get respect from the community

And (hopefully) get many stars on GitHub

How do you make $$$ ?

• A marketplace for OSS and closed source modules

• All modules are certified by Redis Labs for these platforms:

Module Hub

1M+ instances 350K+ instances 100+ F500 Companies

www.redismodules.com

Geo Clustering

Geo Clustering

Adding a Module to Module Hub

Upload

A few clicks

Adding a Module to Module Hub

Upload CertificationProcess

One monthA few clicks

Adding a Module to Module Hub

Upload AlphaCertificationProcess

One monthA few clicks Still not shown in the marketplace.

Test by 10 selected customers

Adding a Module to Module Hub

Upload Alpha BetaCertificationProcess

One month At least 1 month free

A few clicks Still not shown in the marketplace.

Test by 10 selected customers

Adding a Module to Module Hub

Upload Alpha Beta GACertificationProcess

One month Still not shown in the marketplace.

Test by 10 selected customers

At least 1 month free

A few clicks

So – can you make $$$ here ?

Full text search Prefix search0.00

5.00

10.00

15.00

20.00

25.00

30.00

3.15 2.40

21.00

8.70

24.57

10.61

Average Latency (msec)

RLEC Elasticsearch Solr

Full text search Prefix search0

5,000

10,000

15,000

20,000

25,000

20,045

6,831

690

3,686

621

3,133

Ops/sec

RLEC Elasticsearch Solr

85% higher

32x higher

7.8x faster 4.1x faster

redisearch

The world fastest text search engine

For only $0.114/hr

redisearch – the world fastest text search engine

Full text search Prefix search0.00

5.00

10.00

15.00

20.00

25.00

30.00

3.15 2.40

21.00

8.70

24.57

10.61

Average Latency (msec)

RLEC Elasticsearch Solr

Full text search Prefix search0

5,000

10,000

15,000

20,000

25,000

20,045

6,831

690

3,686

621

3,133

Ops/sec

RLEC Elasticsearch Solr

85% higher

32x higher

7.8x faster 4.1x faster

The Possibilities areTruly Endless

Thank You@yiftachsh

Salvatore SanfilippoCreator of Redis

Adrian Colyer Venture Partner at Accel

Author of the Morning Paper BlogFormer CTO, VMWare, Pivotal and Spring Source

As I get older, my memory just keeps getting better and better!

blog.acolyer.org

350Foundation

sFrontiers

A new era of hardware (and what it might mean for Redis)

A new era of data structures?

Talk outline

a

b

Human computers at Dryden by NACA (NASA) - Dryden Flight Research Center Photo Collection

http://www.dfrc.nasa.gov/Gallery/Photo/Places/HTML/E49-54.html. Licensed under Public Domain via Commons - https://commons.wikimedia.org/wiki/File:Human_computers_-_Dryden.jpg#/media/File:Human_computers_-_Dryden.jpg

Computing on a human scale

63

10ns70ns10ms

10s1:10

s116d

Registers & L1-L3

File on desk

Main memory

Office filing cabinet

HDDTrip to the warehouse

ComputeHTMPersistent Memory NIFPGAGPUs

MemoryNVDIMMsPersistent Memory

Networking

100GbERDMA

StorageNVMe

Next-gen NVM

Next generation hardwareAll change please

64

2-10m

Computing on a human scale

65

10s1:10

s116d

File on deskOffice filing cabinetTrip to the warehouse

4x capacity fireproof local filing cabinets

23-40mPhone another office (RDMA)3h20

mNext-gen warehouse

The new ~numbers everyone should know

66

Latency Bandwidth Capacity/IOPS

Register 0.25ns

L1 cache 1ns

L2 cache 3ns 8MB

L3 cache 11ns 45MB

DRAM 62ns 120GBs 6TB - 4 socket

NVRAM’ DIMM 620ns 60GBs 24TB - 4 socket

1-sided RDMA in Data Center 1.4us 100GbE ~700K IOPS

RPC in Data Center 2.4us 100GbE ~400K IOPS

NVRAM’ NVMe 12us 6GBs 16TB/disk,~2M/600K

NVRAM’ NVMf 90us 5GBs 16TB/disk, ~700/600K

NVMe Example: Redis on Flash

67

• 2M IOPS• Sub-ms latency

• 1:4 DRAM:NVM split

(40x SATA SSD)

NVMf Example: E8 Storage

68

$12M Series B led by Accel

• 10M IOPS in single appliance, accessed via RDMA• 100us/50us read/write latency• ...and this is with NAND Flash!

Data tiering in heterogeneous memory systems

DDR-DRAM NVMCapacity per CPU 100s of GBs Terabytes

Read latency 1x 2x to 4x

Write bandwidth 1x ⅛ x to ¼ x

Estimated cost 5x 1x

Endurance 1016 106 to 108

Dulloor et al. 2016 (EuroSys)

What will it mean?

70

this!

All DRAM (baseline)

All NVM

tiered memory, with increasingNVM proportion

Impact on cost?

Impact on performance?

Impact on performance/$?

Key-value store results - performance impact

Proportion of DRAM

Performance relative to all DRAM

NVM only 1.15 - 1.45x

1/8 1.14 - 1.4x

1/4 1.06 - 1.13x

1/2 1.05x

beware!

Key-value store results - cost impact

Proportion of DRAM

Performance/$ relative to all DRAM

NVM only 3.45 - 4.5x

1/8 3.1x

1/4 2.3x

1/2 1.6x

And don’t forget about capacity!

Proportion of DRAM

Performance relative to all DRAM

Performance/$ relative to all DRAM

Capacity relative to all DRAM

NVM only 1.15 - 1.45x 3.45 - 4.5x 4 - 10x

1/8 1.14 - 1.4x 3.1x 3.6 -8.9x

1/4 1.06 - 1.13x 2.3x 3.3 - 7.8x

1/2 1.05x 1.6x 2.5 - 5.5x

74

Ultra low latency, PBs of data

• The RAMCloud storage system - Ousterhout et al. 2015• Implementing linearizability at large scale and low latency - Lee et al. 2015

1000+ Servers

PB+ Memory RDMA RIFL

75

RAMCloud Performance

Operation End-to-end Latency median (99.9%-ile)

Read 4.7 (9.2)μs

Write (durable) 13.4 (148)μs

Transaction 20μs

5-object transaction 27μs

TPC-C throughput (10 nodes) 35K tps

76

No compromisesDistributed transactions with consistency, availability, and performance

Dragojević et al. 2015

“This paper demonstrates that new software in modern data centers can eliminate the need to compromise. It

describes the transaction, replication, and recovery protocols in FaRM, a main memory distributed computing

platform. FaRM provides distributed ACID transactions with strict serializability, high availability, high throughput and low latency. These protocols were designed from first principles to leverage two hardware trends appearing in

data centers: fast commodity networks with RDMA and an inexpensive approach to providing non-volatile

DRAM.”

77

FaRM PerformancePerformance

TPC-C tps (90 nodes) 4.5M

99%-ile latency 1.9ms

K-V Store* qps (20 nodes) 146M

latency (at peak throughput) 35μs

(* FaRM: Fast Remote Memory - Dragojević et al. 2014)

78

Exploiting HTM...

Persist.Memory RDMA HTM DrTM

Fast in-memory transaction processing using RDMA & HTM - Wei et al. 2015

5.5Mtps(TPC-C, 6 node cluster)

Redis is way more than just a ‘K-V store’...

b A new era of data structures?

Evolution of the humble data structure

80

Collection classes& stdlibs

?

single thread/core

Concurrentdata types

multi- thread/core

Distributeddata types

multi-server

Sharedconcurrentdata types

shared-server

PACELC

81

Libraries, Servers, & PACELC

App Instance App InstanceApp InstanceCRDT Library

DS Server DS Server

Client Client Client Client Client

CRDT Server

82

CRDT Evolution

Operation Based

● prepare and effect

● smaller messages

● require reliable exactly-once causal broadcast

● membership management

● no batching

State Based

● state shipping and idempotent join

● larger messages

● support unreliable communication channels

● accommodate batching

Delta-State Based

● local execution, ship representation of the effect & idempotently join

● smaller messages

● support unreliable communication channels

● accommodate batching

Data Structures! E.g. GCounter and PNCounter{ replica_id value }

83

What are CRDTs made of?

{“A”->3, “B”->5}HINCRBY gcounter A 1{“A”->4, “B” -> 5}

HVALS gcounter… 4+5 = 9

{“A” -> 4, “B” -> 7}

{“A”->3, “B”->5}HINCRBY gcounter B 1HINCRBY gcounter B 1{“A”->3, “B” -> 7}

{“A” -> 4, “B” -> 7}

delta B -> 7

delta A -> 4

A B

Causal Context X Dot Store

84

Causal ∂-CRDTs

version vector(Redis hash) Dot.Event

(Redis set or hash)

Dot = replica_id:sequence_no

INCR

+ a delta queue for propagation...

WATCH my_crdt:seq_nonext = INCR my_crdt:seq_no MULTIHSET my_crdt:ccontext replica_id nextSADD my_crdt:dotstore “replica_id:next:val”RPUSH my_crdt:deltaq deltaEXEC

A new era of hardware (and what it might mean for Redis)

A new era of data structures

Summary

a

b

Thank You@adriancolyer

Salvatore SanfilippoCreator of Redis

Laura MerlingFormer VP and GM of IoT Initiative at SAP

The Internet of Things Real Time Reality

The IoT Landscape

PLATFORMS AND ENABLEMENTSOFTWARE, FULL-STACK, DEVELOPER, ANALYTICS, OPEN SOURCE

Big Data with an IOT View

Variability

Velocity

Volume

VarietyComplexity

VelocityData streams and near-real time; RFID tags, sensors and smart metering.

VarietyStructured and unstructured.

VariabilityInconsistent; event triggered.

ComplexityMultiple sources; connect and correlate

relationships, hierarchies and multiple data linkages.

VolumeBillions of sensors and machine to

machine transactions.

Analytics Continuum

DiagnosticDescriptive

PrescriptivePredictive

What Happened Why Did it Happen What Will Happen Make it HappenIn Real Time

Geospatial

The Redis IOT Opportunity

PubSub DatabaseCaching

Time Series

Challenges• Latency• Automation• Complexity

Oil and Gas

Challenges• Transformer• Distribution

Electric Vehicle Charging

Manufacturing

Challenges• Unpredictable • Adherence • Unavailable

Construction

Challenges• Awareness• Congestion• Slow

Connected Home

Challenges• Latency• Interaction• Simplification

Retail

Challenges• Latency• Interaction/Engagement• Inventory

The Data

MINING IoT FOR THE UNFILTERED TRUTH

Thank You@magicmerl

Salvatore SanfilippoCreator of Redis

Reynold XinChief Architect Spark at Databricks

Spark and Redis: In-situ analytics beyond Hadoop

Reynold Xin@rxin2016-05-11 RedisConf

About DatabricksFounded by creators of Spark in 2013

Cloud enterprise data platform- Managed Spark clusters- Interactive data science- Production pipelines- Data governance, security, …

Please put up your hand

if you know what Spark is?

Please put up your hand

if you think your significant otherknows what Spark is?

open source data processing engine built around speed, ease of use, and sophisticated analytics

largest open source data project with 1000+ contributors

“Spark is the Taylor Swift of big data software.”

- Derrick Harris, Fortune

Analytics: a traditional approach

Hadoop is complex to operateData consistency is difficult to maintain

Analytics in-situSQL

ML

Streaming

Analytics in-situSQL

Streaming

MLEnable SQL analytics over RedisUse Redis to store streaming data

Use Redis to serve results generated by Spark

Demoby Richard Garris

Discount code: Meetup16SF

Thank You@rxin

Salvatore SanfilippoCreator of Redis

Enjoy RedisConf

UpdatesYour morning break is located downstairs in the Sponsor Showcase

Three concurrent Breakout Sessions will begin at 11:15am

The Developer Workshop begins at 1:00pm in the Developer Café

Save your Redis Community Raffle ticket for today’s drawingfor a chance to win a Phantom Drone

11:15am-12:00pm New Redis Capabilities: Own, Make, Share

1:00pm-1:45pm SSH I/O Streaming Via Redis-based Persistent Message Queue

2:00pm-2:45pm Walmart and IBM Revisit the Linear Road Benchmark

3:15pm-4:00pm Back your App with MySQL & Redis, the Cloud Foundry Way

4:15pm-5:00pm Postgres and Redis Sitting in a Tree

122

BREAKOUTS: WEDNESDAY, MAY 11

Salvatore SanfilippoCreator of Redis