introducing ibm message hub: cloud-scale messaging based on apache kafka

3499: Introducing IBM Message HubCloud-scale messaging based on Apache Kafka

Andrew Schofield

Chief Architect, Hybrid Cloud Messaging

Organizations must restlessly re-invent themselves

• Outpace existing – and emerging – competitors

• Deliver an increasingly engaging customer experience

• Continuously learn how to improve and anticipate shifts

1

Business success increasingly depends on applying new and innovative

business models and implementing IT solutions more rapidly than

competitors…

Digital IT Enterprise IT

Message Hub(Based on Apache Kakfa)

MQ Light API

MQ Light API2

Enterprise Messaging & Integration

Hybrid Messaging

MQ ExplorerMQ ClientMQ Light API

Kafka API REST API IBM MQ Appliance

Bluemix Public

Bluemix Dedicated(Announced, GA March)

Bluemix Local(Coming soon)

IBM MQ

Cloud

On-Prem

Cloud

On-Prem

IBM MessageConnect

What is “Digital IT” anyway?

3

Speed and agility to drive innovation and growth•Explore, adopt, adapt•Rapid, iterative prototypes

Valuable

business

assets

Exposed as

APIs

Consumed

by app

developers

To develop

innovative apps

Delightful

experience to

customers

Feedback loop enabling

rapid evolution

Or delivered

as events

IBM Bluemix

4

Bluemix is an open-standards, cloud-based platform for building,

running, and managing applications

Build your apps, your way

Use the most prominent compute

technologies to power your app:

Cloud Foundry, Docker,

OpenStack.

Extend apps with services

A catalog of IBM, third party, and

open source services allow the

developer to stitch an application

together quickly.

Scale more than just instances

Development, monitoring,

deployment, and logging tools

allow the developer to run and

manage the entire application.

Layered security

IBM secures the platform and

infrastructure and provides you with

the tools to secure your apps.

Deploy and manage hybrid apps

seamlessly

Get a seamless dev and

management experience across a

number of hybrid options.

Flexible pricing

Try compute options and services

for free and, when you’re ready, pay

only for what you use. Pay as you

go and subscription models offer

choice and flexibility.

Message Hub is…..

• A scalable, distributed, high-throughput message bus based on Apache Kafka

• Wide compatibility via 3 APIs : REST, Kafka and MQ Light API (coming soon)

• Tightly integrated with the Bluemix platform

MQ (plus other on-prem data sources)

MQ Light API

AMQP

RESTful API

HTTP

Apache Kafka

TCPStreams

…

Bluemix Public

Bluemix Dedicated(Announced, GA March)

Bluemix Local(Coming soon)

Available for …

5

IBM MessageConnect

What is Apache Kafka?• Apache Kafka is publish-subscribe messaging re-thought as a

distributed commit log. The following attributes are claimed of Kafka:

• Fast - a single broker on commodity hardware can handle hundreds of megabytes of reads and writes per second from thousands of clients.

• Scalable - a single cluster can serve as the central data backbone for a large organization. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine.

• Durable - messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact.

• LinkedIn developed Kafka to solve its internal data integration problems

• Felt available commercial solutions didn’t address their

scalability requirements

• Open sourced early 2011

• Apache incubator graduation fall 20126

Why Apache Kafka?

7

Fast: Responds to your needs

Scalable: Scales according to demands

Durable: Prevents data loss

through replication

Distributed: Fault-tolerant

Used by companies

including:

LinkedIn

Yahoo

Twitter

Netflix

Spotify

Pinterest

Paypal

Tumblr

Anatomy of a topic

Kafka topics are made up of multiple partitions– Partitions are distributed across Kafka nodes

Partitions are ordered streams of messages– Replicated copies distributed across Kafka nodes

T0P0

T0P0

T0P1

T0P1

T1P1

T1P1

T1P0

T1P0

Kafka Node 1 Kafka Node 2 Kafka Node 3 Kafka Node 4

M0 M1 M2 M3

8

What does Message Hub enable?

9

Hub for asynchronously

connecting services

inside Bluemix or beyondApplications connected to events

happening in other Bluemix

services, or from beyond the

cloud

Insights from the data you already haveData needs to be streamed from anywhere to one or many

analytics engines

1 2

3

Microservices allow

applications to evolve

rapidlyOpen protocols support

polyglot runtimes,

application controlled

behavior and reactive scale

MQ

plus other

on-prem data sources

4


10


connecting services




cloud

MQ

plus other


Speeds production: Messy

wiring between services taken

care of for you

Offers a range of

solutions: Choose the

services that you need, and

use Message Hub for the

messaging

Connects with enterprise on-

premise: All the benefits of

working in a hybrid environment

1

Getting messages from one service to another

11

Think of Message Hub as the conduit between your

cloud services….

…taking data, in the form of messages, from one

service to another


12

2 Microservices allow



polyglot runtimes,



Innovate and adapt quickly: Deploy and scale microservices

independently

Flexibility: Work in a range of

languages that suit you

Reliability: Remove the

interdependency between

microservices

One day in a world without microservices…

13

(Except he has to re-build the entire app, because of

the way that it has been built, so it’s not so quick…)

Chris, your developer, decides to

make a quick change to the app

His change introduces an

error, and the whole thing

crashes

A supermarket runs an app that

allows customers to take

advantage of special offers

when they are in specific areas

of the store

And with microservices?

14

Working in a microservices

framework means that

applications are broken into

smaller parts

And because they are independent

So that changes to individual parts

can be quickly made

One change does not affect the

other parts

The same scenario with microservices

15

IT’S QUICK because

he doesn’t have to rebuild

the entire app

OK, his code breaks, but

the rest of app is

unaffectedQUICK TO FIX because he

doesn’t have to rebuild the entire app

Chris, your developer, decides to

make a quick change to the app

A supermarket runs an app that

allows customers to take

advantage of special offers

when they are in specific areas

of the store

MQ Light API – Messaging for app developers

• Messaging that application developers will love to use, helping them make responsive applications that scale easily

• 3 ways to get it:

– MQ Light software download for developers

– IBM MQ 8.0.0.4 and later

– Message Hub in Bluemix

• Open APIs crafted to feel natural in a growing range of popular languages

• Tooling that makes modular app development easy

16


17


analytics engines

3

Adapt to change: React to

changing trends as they happen

Consume data in your way: Run real time and batch analytics

on the same data Protection of valuable data: Acts as a buffer between your

data and the analytics engine

Imagine the possibilities…

18

Fraud andrisk

detection

Real-time traffic flow

optimization

Accurate and timely threat

detection

Understand and act on customer sentiment

Predict and act on intent to purchase


19


connecting services




cloud


analytics engines

1 2

3

Microservices allow



polyglot runtimes,



MQ

plus other


4

Message Hub delivery strategy

20

Message HubMessage Hub

IncubatorFeatures & capabilities

Promoted when ready

• MQ Light API

Future:

• …

• [continuously delivered

new features]

Introducing Message Hub Incubator

• A separate, experimental service whose content changes over time

– Find it in the Bluemix Labs catalog: https://console.ng.bluemix.net/catalog/labs/

• Early access to forthcoming Message Hub features

– Under active development so we need freedom to make significant changes

– We reserve the right to discard data, so we don’t need to handle migration as we develop

• The first delivery of MQ Light API support

– Full MQ Light API support for Java and Node.js (but not Ruby) clients

– Limited infrastructure behind the scenes

• Come and try it out in Lab 3508: IBM MQ Light – Cloud, Developer and Enterprise

– MGM Grand, Room 319 – Wednesday – 1:15PM to 2:15PM

21

AMQPAdapter

Message Hub architecture

ApacheKafka

Cluster

HTTP Load Balancer

MQ Light API

AMQP

RESTful API

HTTP

Apache Kafka

TCP

Topic

mapper

ZooKeeper

AMQPAdapter

(stateless)

AMQPAdapter

AMQPAdapter

AMQPAdapter

KafkaREST

Interface

AMQP State Store

Service Lookup

22

Performance

• Throughput of the cluster is about 300,000 msg/s

– 100-byte messages

– Secure connection from public network

– Disk encryption of Kafka log

– Secure connections for all user data, even within DMZ

• Latency ranges from 20ms to 100ms, average 50ms

23

Continuous delivery and microservices

• Message Hub is developed using a continuous delivery process and a microservice architecture

– It’s built out of small pieces that we can enhance rapidly

• Multiple deployment environments with promotion from dev/test towards production

– We can deploy changes into production as often as required, even several times as day if we had to

• When there’s a critical patch for Kafka, we…

– Build the patch as deploy to dev/test

– Test it thoroughly

– Evaluate suitability for promotion to staging…

– And repeat until the patch reaches production

24

Message Hub continuous delivery pipeline

Build Unit TestDocker Image

Promotion process

Dev-test

Staging

Bluemix (internal)

Bluemix (public)

Marathon / Mesos / Docker

Jenkins CI Pipeline

Integration testing

“Chaos Monkey”

Load testing

25

Involvement in the community

• Contributed several patches, particularly as release of 0.9 approached

– Primarily related to SSL/TLS and SASL

• Proposed a Kafka Improvement Proposal to augment SASL capabilities

– KIP-43

• Started discussion on exactly-once delivery to improve inter-protocol bridging

• Have an accepted session for the Kafka Summit in April 2016

26

Why Apache Kafka as a service?

27

Option 1

• Deploy your own Kafka cluster

• Deploy Zookeeper cluster to support it

• Put systems in place to operate the service 24x7

• Keep up-to-date with the open source community

• Handle version-to-version migration and updates

Option 2

• Use Message Hub

See our 3-minute video

showing how to get

started!

IBM Message Hub is live!

28

A scalable, distributed, high throughput message bus for your cloud-native applications

Connects services inside Bluemix

with your systems beyond

Connects microservices,

using open-standard protocols

Streams data to analytics for

real-time insights

Find us in the Bluemix catalog and sign up at

https://console.ng.bluemix.net/catalog/message-hub/

Built on Apache

Where do I get more information?

IBM Messaging developerWorks

developer.ibm.com/messaging

IBM Messaging Youtube

https://www.youtube.com/IBMmessagingMedia

LinkedIn

Ibm.biz/ibmmessaging

Twitter

@IBMMessaging

IBM MQ Facebook

Facebook.com/IBM-MQ-8304628654/

Monday

10:30-11:30 3592 New MQ features3452 Managing applications

12:00-13:00 2835 MQ on z/OS and Distributed

15:00-16:00 3470 Latest MQ z/OS features2833 Where is my message?3544 MQ Light in an MQ infrastructure

16:30-17:30 3573 Hybrid cloud messaging2941 MQ Advanced

Tuesday

08:30-09:30 3540 The MQ Light API

12:00-13:00 3456 The IBM MQ Appliance

13:15-14:15 3499 Introducing Message Hub3458 MQ Appliance administration

14:30-15:30 6432 MQ updates and futures (InnerCircle)

2849 Messaging feedback roundtable

16:00-17:00 3544 MQ Light in an MQ infrastructure3513 MQ hands on lab

Wednesday

08:30-09:30 3602 Effectively Managing Hybrid Messaging

12:00-13:00 3613 Designing MQ self service6408 Hybrid messaging roadmap (InnerCircle)

13:15-14:00 3416 HA and DR with MQ3433 Why secure your messaging?

15:45-16:30 3429 Securing MQ2847 Meet the messaging experts

16:00-17:00 3508 MQ Light hands on lab

16:45-17:30 2275 Migrating to the IBM MQ Appliance

Thursday

08:30-09:15 3420 MQ Clustering2931 Business agility with self service MQ

09:30-10:15 3479 MQ z/OS clusters and shared queue3450 Optimising MQ applications2849 Messaging feedback roundtable

10:30-11:15 3465 MQ Appliance high availability3481 MQ z/OS messaging connectivity

11:30-12:15 3474 Active-active messaging3537 Monitoring and managing MQ3425 MQ publish/subscribe

Find us at the EXPO:

Hybrid Integration peds 65-68

Check out the Hybrid Messaging sub topic under the

Hybrid Integration topic for further customer and business

partner sessions

Hybrid Messaging from the IBM experts at InterConnect 2016 Sunday

14:30-15:30 6408 Hybrid messaging roadmap (InnerCircle)

• Hybrid Integration Strategy

• Cloud Integration

• Accelerating Digital Business

• Integration Bus

• IBM MQ

• API Management

• BPM / ODM

• DataPower

• CICS

• WASSpend time with IBM experts, at the home of many of IBM's software products. This summit is by

invitation only - a limited seating engagement for executives and architects who would like to learn how to

harness IBM connectivity and application integration solutions to deliver access to data, applications and

information regardless of platform, device or data formats - across both on-premises and cloud

environments.

Learn more about how we are transforming our technologies using Hybrid Cloud to enable you to harness

your existing assets to achieve greater capacity, efficiency and integration across platforms, whilst retaining

the security, capability and resiliency you would expect from IBM.

• Discover and influence IBM's strategy for key messaging and integration technologies, including, IBM

MQ, IBM Integration Bus and IBM API Management

• Engage in technical sessions and one-on-one interactions with top IBM Hursley Lab architects and

senior executives to refine your 2016 strategic plans

• Expand your network with industry-leading peers from other companies

• Plus learn about other IBM technology, such as IBM intelligent business process management solutions

(BPM & ODM), DataPower gateways, CICS and WebSphere Application Server on-premise and cloud

This event is conducted under a Non-Disclosure agreement, so we will be able to share product

directions with you.

Hursley: a visit to talk about

The IBM Hursley Lab is the largest

softw are development facility in Europe;

situated in a beautiful 100 acre park w ith a

historic setting. Attendees stay in the local

city of Winchester w hich is a vibrant

heritage destination w ith many attractions

and classical architecture including a

magnificent cathedral.

Enjoy the aw ard-winning pubs and

restaurants and a tempting array of

independent shops.

Talk to your IBM rep to find out more

Be part of the conversation

Keep up to date with the latest information, join the conversations

and help to shape the event to meet your interests. Use

#IBMhursum in your Tweets to keep in touch.

#IBMhursum

European & North American

Hursley Summit 2016Integration across applications, data and processes for mobile and cloud

May 10 – 12 & May 16 - 19 | IBM Hursley Lab, UK#IBMhursum

Notices and Disclaimers

32

Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law

Notices and Disclaimers Con’t.

33

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.

IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

Thank YouYour Feedback is Important!

Access the InterConnect 2016 Conference Attendee

Portal to complete your session surveys from your

smartphone,

laptop or conference kiosk.