Download - Unified log-meetup-20160420
![Page 1: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/1.jpg)
Introducing IBM Message Hub
Oliver Deakin20/04/2016
Apache Kafka as a Service
![Page 2: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/2.jpg)
2
To do…
What is IBM Bluemix? What is Message Hub? What is Message Hub… really? What else have we done?
![Page 3: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/3.jpg)
3
What is IBM Bluemix?
Bluemix is an open-standards, cloud-based platform for building, running, and managing applications
Less managed
More managed
![Page 4: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/4.jpg)
4
What is IBM Bluemix?
![Page 5: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/5.jpg)
5
To do…
What is IBM Bluemix? What is Message Hub? What is Message Hub… really? What else have we done?
![Page 6: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/6.jpg)
6
What is Message Hub?
![Page 7: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/7.jpg)
7
What is Message Hub?
![Page 8: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/8.jpg)
8
To do…
What is IBM Bluemix? What is Message Hub? What is Message Hub… really? What else have we done?
![Page 9: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/9.jpg)
9
What is Message Hub… really?
Kafka 0.9
Some additions– Multi-tenancy– Security– Confluent REST API– Administration REST API– Monitoring
![Page 10: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/10.jpg)
10
Message Hub is… Kafka
Kafka 0.9– Took early releases– Built, tested and deployed– Deployed as Docker containers– Continuous delivery pipeline
Community involvement– Active submitting bug fixes– KIP-43 – extend SASL mechanism support
![Page 11: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/11.jpg)
11
Message Hub is… Kafka
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
![Page 12: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/12.jpg)
12
Message Hub is… Multi-tenant
Single Kafka clusterSupport up to 500 tenantsRestrictions on each tenant’s resources
– Up to 100 partitions per tenant (3-way replicated)– Max partition size 1G– Max message age 24 hours– Clean-up policy is deletion– Max message size 1M
![Page 13: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/13.jpg)
13
Message Hub is… Multi-tenant
Kafka
Zookeeper
User1 App
Send to mytopic
User2 App
Send to mytopic
User1 clientLookup User1
tenant prefix, 1234
Add tenant prefix to mytopic
Continue with topic name 1234mytopic
User2 clientLookup User2
tenant prefix, ABCD
Add tenant prefix to mytopic
Continue with topic name ABCDmytopic
![Page 14: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/14.jpg)
14
Message Hub is… Secure
TLS introduced in Kafka 0.9Clients authenticated with SASL Plain IBM cloud security standards
– Customer message data encrypted everywhere– Data storage must be separated from public interface
![Page 15: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/15.jpg)
15
Message Hub is… Secure
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
![Page 16: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/16.jpg)
16
Message Hub is… Secure
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
TLS
TLS
TLS TLS
TLS
![Page 17: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/17.jpg)
17
Message Hub is… Secure
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
TLSTLS TL
S TLS TLS
HA Proxy HA Proxy HA Proxy HA Proxy HA Proxy
![Page 18: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/18.jpg)
18
Message Hub is… Secure
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
TLSTLS TL
S TLS TLS
HA Proxy HA Proxy HA Proxy HA Proxy HA Proxy
TLS
TLS
TLS
TLS
TLS
![Page 19: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/19.jpg)
19
Message Hub is… Secure
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
TLSTLS TL
S TLS TLS
HA Proxy HA Proxy HA Proxy HA Proxy HA Proxy
TLS
TLS
TLS
TLS
TLS
![Page 20: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/20.jpg)
20
Message Hub is… Secure
Throughput of 900k msgs/s– 100 byte messages– 6 clients– 1000 topics @1 partition
Latency ranges from 10 – 70ms typically
![Page 21: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/21.jpg)
21
Message Hub is… Flexible
Kafka REST Proxy– Confluent platform 2.0– Binary message format only– No schema registry
![Page 22: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/22.jpg)
22
Message Hub is… Flexible
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
RESTREST
HA Proxy HA Proxy HA Proxy HA Proxy HA ProxyHA Proxy
![Page 23: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/23.jpg)
23
Message Hub is… Managed
Kafka Administration REST interface– Used to manage topic lifecycle– Topics not auto-created– Web UI also available for manual management
Topics can be created and deleted– Topics can have 1+ partitions– 100 partition limit per tenant across all topics
![Page 24: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/24.jpg)
24
Message Hub is… Managed
Monitoring… and lots of it!– collectd containers deployed on every host– metrics pushed locally– forwarded to central graphite + grafana dashboards– continuous testing
Automated alerts– thresholds trigger support callouts– Slack integration to notify whole team
![Page 25: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/25.jpg)
25
Message Hub is… Managed
![Page 26: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/26.jpg)
26
To do…
What is IBM Bluemix? What is Message Hub? What is Message Hub… really? What else have we done?
![Page 27: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/27.jpg)
27
Message Connect
Connect Enterprise and Cloud– Experimental Bluemix service– Messaging pipeline from MQ to Message Hub– Bluemix apps have visibility of on-prem messages– Allows connectivity with cloud analytics and other services
![Page 28: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/28.jpg)
28
Message Hub Incubator
AMQP proxy layer– Experimental Bluemix service– Kafka is backing messaging engine– Horizontally scalable
Target is to provide interoperability with Kafka clients
![Page 29: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/29.jpg)
29
Bluemix Dedicated and Local
Bluemix Dedicated– IBM cloud and IBM managed– Customer gets dedicated Bluemix environment– Full Message Hub cluster– Spec defined by customer requirements– Available now
Bluemix Local– Customer’s cloud but IBM managed– TBD…
![Page 30: Unified log-meetup-20160420](https://reader035.vdocuments.us/reader035/viewer/2022070603/587285001a28abc7068b6f29/html5/thumbnails/30.jpg)
30
Twitter: @IBMmessaging
DeveloperWorks:https://developer.ibm.com/messaging/message-hub/
Bluemix blogs:https://developer.ibm.com/bluemix/blog/
LinkedIn: ibm.biz/ibmmessaging
Message Hub service:https://console.ng.bluemix.net/catalog/services/message-hub