Berlin Apache Flink Meetup #12Community Update & 2015 – The Year in Review

December 2015

Robert MetzgerCommitter and PMC

Memberrmetzger@apache.org

@rmetzger_

2

What happened?• 0.10.0 and 0.10.1 released– Event-time streaming (out of order streams,

sessions)– Master HA no single point of failure– DataStream API graduated– New monitoring interface– Offheap memory– Outer joins– Gelly Scala API

• Otto released a Flink testing framework: https://github.com/ottogroup/flink-spector

flink.apache.org

3

Now in master (1.0-SNAPSHOT)

flink.apache.org

• Kafka Connector improvements: Multiple topics, keys, deleted messages, periodic offset commit

• Flink in firewalled setups• Out of core state backend for JDBC• Major improvements in the flink-storm

module• RabbitMQ Source with exactly-once• Many other operational and usability

improvements

4

Pending Pull Requests• Savepoints: Trigger state checkpoints

manually and restore a job from a savepoint

• Submit Flink jobs through JobManager web interface

• Add range partitioner with sampling (FLINK-7)

• “stop” signal for streaming topologies• Monitoring: JMX beans & checkpoint metrics

flink.apache.org

5

Reading List• Realtime dashboards with Flink and ElasticSearch

https://www.elastic.co/blog/building-real-time-dashboard-applications-with-apache-flink-elasticsearch-and-kibana

• Introduction to streaming windowshttp://flink.apache.org/news/2015/12/04/Introducing-windows.html

• Flink + Storm: Run Storm topologies on Flinkhttp://flink.apache.org/news/2015/12/11/storm-compatibility.html

• New streaming use caseshttp://data-artisans.com/how-apache-flink-enables-new-streaming-applications-part-1/

• Streaming with Flink Intro (reading twitter data) http://blog.brakmic.com/stream-processing-with-apache-flink/

flink.apache.org

6

Flink 2015: The journey to mainstream

2015 – The Year in Review

flink.apache.org

7

Let the numbers speak: GitHub

flink.apache.org

280%+

Stars 340%+

Forks

February 2015

December 2015

8

Let the numbers speak: Meetup Groups

flink.apache.org

From one Meetup group to a global community

9

Let the facts speak:Organizations at Flink Forward 2015

flink.apache.org 9

Let the facts speak:Articles about Flink featured in

10

Let the facts speak:Hadoop Summit 2016: 17 Flink talks

11

12flink.apache.org

• Check out the slides: http://flink-forward.org/?post_type=session

• Video recordings on YouTube, “Flink Forward” channel

Apropos

13

Feature Timeline 2015• January

– Flink 0.8• February

– Gelly– Table API

• March– Flink-Google Data

Flow Runner (batch)– FlinkML

• April– Python API

• May– Static Code Analysis– Zeppelin Integration

• June– 0.9: Akka, batch

exchange, exactly-once

• July– New dashboard– first HA features

flink.apache.org

14

Feature Timeline 2015• August

– Gelly Scala API– DataSet.sample()

• September– Cascading on Flink– Scala 2.11 (and 2.10)– Offheap memory– Reworked windowing– Rolling file sink– Elasticsearch sink– High Availability

flink.apache.org

• October– Reworked Kafka

Consumer– Streaming API

graduation• November– Flink 0.10– JDBC backend

• December– Flink behind firewalls

15

git Stats 2015git log --pretty=oneline --after=1/1/2015 | wc -l

2201 commits in 2015

flink.apache.org

Lines of code Commits per day

Commits per hour

16

We are not done yet• Flink Forward 2016• Roadmap 2016 (pending community discussion)

– Autoscaling– Stream SQL–Managed Memory State… much more

Let’s make 2016 the Year of Flink!flink.apache.org