samza tech talk_2015 - huawei

Stream Processing @Scale in LinkedIn

Yi PanData Infrastructure

Samza Team @LinkedIn

Databus

• What is Stream Processing?• What is Samza?• Samza Programming API• Stream Processing @LinkedIn• Upcoming features

Overview

• What’s stream processing– Input: an unbounded sequence of events

• E.g. web server logs, user activity tracking events, database changelogs, etc.

– Latency: near real-time• From milliseconds to minutes, instead of hours to

days– Output: an unbounded sequence of changes to

the derived dataset• The derived dataset is usually the final or partial

analytic results that can either be in another stream, or a serving data store

Stream Processing

Response latency

Stream

Processing

Milliseconds to minutes

Synchronous Later. Possibly much later.

Stream Processing

• What are the application requirements?– Scalable, fast, stateful stream processing– What scale should we operate at?

• Traffic Volume: 1.4 Trillion events/day• Intermediate State Size: multi TB / colo (*)

– Why is it expensive to run stream processing at scale?

• Intermediate data set needs to be stored to allow low latency processing

• Large volume of data needs to be pulled and pushed via network

Stream Processing

Overview

• Samza is a distributed Turing machine– Single Task Samza Job is a stateful

Turing machine

What’s Samza

Samza TaskInput stream Output stream

Statechangelogch

– Scaling a Samza job: partition the streams

What’s SamzaIn

partition 0

partition 1

partition 2

partition 3

partition n

Samza Task

– Scaling a Samza job: partition the streams

What’s SamzaIn

partition 0

partition 1

partition 2

partition 3

partition n

Samza Task

– Scaling a Samza job: replicating the state machine

What’s Samza

shared checkpoint

• Samza Execution in Yarn

What’s Samza

Host 1 Host 2 Host 3

Application Master

Samza container Samza container

Samza container

Deploy Samza job

What’s Samza

Application Master

Samza container Samza container

Samza container

What’s Samza

Application Master

Samza container

• States in Samza– Checkpoints

• Offsets per input stream partitions– State Stores

• In-memory or on-disk (RocksDB) derived data set

What’s Samza

Samza TaskOutput stream partitions

State changelog partitionsch

Host 1

• States in Samza– Checkpoints and local state stores are backed

by distributed logs

What’s Samza

Host 1

• States in Samza– Checkpoints and local state stores are

backed by distributed logs

What’s Samza

Host 1

• States in Samza– Checkpoints and local state stores are

backed by distributed logs

What’s Samza

Host 2

• Multiple Jobs in a Dataflow

What’s Samza

Stream A Stream B Stream C

Stream E

Stream F

Job 1 Job 2

Stream D

Overview

Partition 0

class PageKeyViewsCounterTask implements StreamTask { public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) {

GenericRecord record = ((GenericRecord) envelope.getMsg()); String pageKey = record.get("page-key").toString(); int newCount = pageKeyViews.get(pageKey).incrementAndGet(); collector.send(countStream, pageKey, newCount); }}

Samza Programming API

Partition 0

Overview

Stream Processing @ LinkedIn

WebServersWebServers

WebServersMonitorServers

Oracle

Espresso

Kafka Databus

Trackingevents

Metrics

changelog

Samza JobsSamza

JobsSamza JobsSamza

bootstrap

VoldemortDerivedData

DerivedData

• Tracking aggregate/analysis (ACG)

• Content standardization w/ adjunct data setMember

Profile DBBootstrap

JobDatabus

Content Standardization

• Kafka Deployment– 1.1 Trillion messages / day

• Databus Deployment– 300 Billion messages / day

• Samza Deployment– multiple colos– 10+ Yarn clusters– 200+ nodes– 100+ Jobs in production

• What is Stream Processing?• What’s Samza• Samza Programming API• Stream Processing @LinkedIn• Upcoming features

Overview

• New features– Local state store improvements

• RocksDB TTL support• Fast recovery

– Dynamic configuration– Easier deployment w/ standalone jobs– High-level query language for faster

development

Upcoming Features

Contact Us / Get Involved• Open Source

–Documentation: samza.apache.org–Mailing list:

[email protected]– JIRA: https

://issues.apache.org/jira/browse/SAMZA

samza tech talk_2015 - huawei

Software