introduction to cassandra • why spark - apache cassandra | apache kafka | apache spark · 2017....

24

Upload: others

Post on 17-Sep-2020

55 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background
Page 2: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

• Introduction to Cassandra

• Why Spark + Cassandra

• Problem background and overall architecture

• Implementation process & lessons learned

• What’s next?

Page 3: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

• Highly available• Master less• Linear scalability• Low latency

• No join• Poor index• Restricted filtering• No ACID

• OLTP• Data ingestion• Design your requests first, your model second.

Page 4: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

Client

Cassandra is a Distributed Hash Table

Assume Replication Factor of 3

Page 5: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

4

Sensor Id, Date, Timestamp, metrics1, ..

Page 6: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

Spark is a Distributed Big Data Processing Framework

Worker + Master (standby)

Worker + Master (leader)

Worker + Master (standby)

Worker

Worker

Page 7: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

Spark Cassandra connector

val rdd = sc.cassandraTable(“my_keyspace", “my_table")

Page 8: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background
Page 9: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

Page 10: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

••••••

••••

Page 11: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

Managed Node

(AWS) x many

Managed Node

(Azure) x many

Managed Node

(SoftLayer) x many

Cassandra + Spark

(x27)

Riemann(x3)

RabbitMQ(x2)

Console/API(x2)

Admin Tools

1400 nodes * ~2,000 metrics / 20 secs = 140k metrics/sec

PagerDutyManaged

Node(GCP) x

many

Page 12: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

1.2.3.4.

~ 9(!) months (with quite a few detours and distractions)

Page 13: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

Managed Node

(AWS) x many

Managed Node

(Azure) x many

Managed Node

(SoftLayer) x many

Cassandra + Spark

(x15)

Riemann(x3)

RabbitMQ(x2)

Console/API(x2)

Admin Tools

1400nodes * ~2,000 metrics / 20 secs = 140k metrics/sec

PagerDutyManaged Node

(GCP) x many

Page 14: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

•••

•••

••

••

Cassandra + Spark

(x15)

Riemann(x3)

Page 15: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background
Page 16: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background
Page 17: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

••••

val RDDJoin = sc.cassandraTable[(String, String)]("instametrics" , "service_per_host") .filter(a => broadcastListEventAll.value.map(r => a._2.matches(r)).foldLeft(false)(_ || _)) .map(a => (a._1, dateBucket, a._2)) .repartitionByCassandraReplica("instametrics", "events_raw_5m", 100) .joinWithCassandraTable("instametrics", "events_raw_5m").cache()

Cassandra + Spark

(x21)

Page 18: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background
Page 19: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

Read Tuning:spark.cassandra.input.fetch.size_in_rowsspark.cassandra.input.reads_per_sec

Write Tuning:spark.cassandra.output.throughput_mb_per_sec

5min – hourly – daily rollup

Page 20: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

Page 21: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

• val RDDJoin = sc.cassandraTable[(String, String)]("instametrics" , "service_per_host") .filter(a => broadcastListEventAll.value.map(r => a._2.matches(r)).foldLeft(false)(_ || _)) .map(a => (a._1, dateBucket, a._2)) .repartitionByCassandraReplica("instametrics", "events_raw_5m", 100) .joinWithCassandraTable("instametrics", "events_raw_5m", SomeColumns("time", "state", FunctionCallRef("avg", Seq(Right("metric")), Some("avg")), FunctionCallRef("max", Seq(Right("metric")), Some("max")), FunctionCallRef("min", Seq(Right("metric")), Some("min")))).cache()

Page 22: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

••

•••

Page 24: Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

[email protected] www.instaclustr.com @instaclustr

• Managed service• Consulting service• Enterprise support contract