building a real-time data pipeline with spark, kafka, and python

Download Building a Real-Time Data Pipeline with Spark, Kafka, and Python

Post on 15-Apr-2017



Data & Analytics

1 download

Embed Size (px)


  • Douglas ButlerProduct Manager

  • massively parallel, lock free, FASTdistributed SQL database

    in-memory, on-diskACID

    JSON and geospatialtransactions and analytics

  • 2 Minute Install

  • A Simple Pipeline

  • from pystreamliner.api import Extractor

    class CustomExtractor(Extractor): def initialize(self, streaming_context, sql_context, config, interval, logger):"Initialized Extractor") def next(self, streaming_context, time, sql_context, config, interval, logger): rdd = streaming_context._sc.parallelize([[x] for x in range(10)]) return sql_context.createDataFrame(rdd, ["number"])

  • > memsql-ops pip install [package]

    distributed cluster-wide

    any Python package

    bring your own

  • Real-time pipeline

  • Q & A time