1
InterConnect2017
HHI-2151Effective Administration of IBM Integration Bus
Sanjay Nagchowdhury
IBM Integration [email protected]
2
Effective Administration…
Installation
Configuration
MonitoringDeployment
Migration
Security
3
Effective Administration…
Installation
Configuration
Deployment
Migration
Monitoring
Security
4
Installation
Radically Simplified Packaging and Installation– Full function, simple, single package install– Developer Operating Systems contain Toolkit and Server
• Total size approx. 1.3 GB– Server Operating Systems contain only server
Other changes– Full entitlement to MQ remains– MQ no longer packaged
• Default queue manager for IB node for backwards compatibility
Built-in Unit Test Environment– Developer tools have built-in unit test server
• Fixed name of TESTNODE_<userid>– Started and stopped with tools– Can still test / deploy to manually created local and remote servers
Single install package for server and toolkit.
Simple unzip on unix.
Local integration node and server automatically created
when you start toolkit.
tar –xzf…
New Tutorials Gallery.
5
Effective Administration…
Installation
Configuration
Deployment
Migration
Monitoring
Security
6
Configuration
TOOLKIT• Connect to local or remote
Integration Nodes using toolkit• Configure bar file properties.• Deploy bar files• Configure policy sets for web
services
CONSOLE• New command console• iib help, iib tools, etc• Remote administration commands
need to point at the integration node’s administration port instead of queue manager.
Integration API• Java interface for administering IIB
through Java code.
REST API• Administer IIB resources using
operations defined on a REST model.
WEB UI• Primary graphical means of
administering IIB.• Many new features in V10.
• Manage Integration Servers
• Define policy• Create configurable
service• View Statistics• Workload Management
7
Manage Integration Servers• Start, stop• View, Create, Rename, Delete
Manage deployed resources• Start, stop, • View, manage, delete
Start/stop statistics and accounting data
Start/stop resource statistics
Start/stop flow monitoring for publishing events, record and replay and business transaction monitoring
Configuration
NEW !
8
Loopback Request Node
• The LoopBack Request node provides a way for IIB message flow authors to invoke synchronous CRUD operations to external systems using LoopBack Connectors.
• LoopBack provides numerous connectors to access enterprise and other backend data systems such as:• NoSQL databases.• Relational databases.
Cloudant
NEW !
Windows, Linux x86
Integration ServerNode.js®
MongoDBConnector
PostgreSQLConnector
CloudantConnector
Non DatabaseConnectors
Community Connectors
LoopBack
Cloudant
Data SourcesIntegration Node
MongoDB
9
ConfigurationNEW ! New LoopBack Request node provides a
way for IIB message flows to invoke synchronous CRUD operations to external systems using LoopBack Connectors.
LoopBack is an open source Node.js framework that can be extended by downloading and installing connectors from a large open source catalog using the ‘npm’ tool.
LoopBack provides connectors for:• NoSQL databases e.g. MongoDB, IBM
Cloudant• Relational databases e.g. PostgreSQL,
MySQL• Other backend services e.g. REST,
SOAP
10
LoopBack Connectors
Contributed by LoopBack
Community
http://loopback.io/doc/
11
Configuration
MQSI_WORKPATH\connectors\loopback\datasources.json
MQSI_WORKPATH\node_modules1
2
NEW !
3 mqsisetdbparms –n loopback::<secid> -u uid –p pwd
npm install loopback-connector-mongodb --save
12
Effective Administration…
Installation
Configuration
Deployment
Migration
Monitoring
Security
13
Toolkit
BAR files contain configurable properties which
allow you to deploy your applications from your QA
system to Production.
Web UI
Message node properties are configurable.
Workload management properties are configurable.
14
Deployment When a REST API is deployed, the Swagger document for that REST API is automatically
made available over HTTP from the same server and port that the REST API is hosted in.
The deployed Swagger document is automatically updated to reflect the server, port,
and HTTP/HTTPS details for the deployed REST API. You do not have to update it with the
correct details before deployment.
The REST APIs can be pushed to API Connect where you can secure them, define Service Level Agreements and
examine visual analytics data.
NEW !
How many Integration Servers should I have?How many Additional Instances should I add?
Additional Instances
• Results in more processing threads• Low(er) memory requirement• Thread level separation• Can share data between threads• Scales across multiple servers
Integration Servers
• Results in a new process/address-space• Increased memory requirement• Multiple threads including management• Operational simplicity• Gives process level separation• Scales across multiple servers
Recommended Usage
• Check resource constraints on system• How much memory available?• How many CPUs?
• Start low (1 server, No additional instances)• Group applications in a single integration server • Assign heavy resource users to their own integration server• Increment integration servers and additional instances one at a time
• Keep checking memory and CPU on machine• Don’t assume configuration will work the same on different machines
• Different memory and number of CPUs 15
Deployment Considerations Deployment
How many copies of a message flow do I need ?
Each message flow is different…..Each will use certain different level of resources (CPU, memory, I/O) and have a particular performance profile
In deciding number of copies to run need to know1. Availability Requirements2. Target throughput (messages/second)
rate3. Target response time
Number of copies of each message flow needed will vary.So do not decide the same number of copies are sufficient in all cases.
Ultimately need to balance resource manageability &
availability.
16
Effective Administration…
Installation
Configuration
Deployment
Migration
Monitoring
Security
17
Monitoring
Workload management policy offers dynamic control over:
• Message rate limits.• Additional instances.• Commit count & commit interval.• Automatic restart in case of
unresponsive message flow.
View message flow and resource statistics in the Web UI. Use this to analyse bottlenecks in your flows.
18
IBM Cloud Product Insights is a Bluemix service that is part of IBM Connect to Cloud.
MonitoringNEW !
20
Different IBM products can be registered, so you can create a cross-product inventory and
view product usage metrics across your portfolio of IBM products in your solution.
Monitoring
If you click on Register a product, you can see which IBM products are available for
registration.
21
You will see all products that are available for registration.
If you click on any of these, you will see instructions on how to register
your product.
22
You can see what is the minimum supported version of IIB.
Links are provided to the product web page and knowledge center.
Simple instructions on how to activate the Bluemix reporting
feature for your Integration Node and Integration Servers.
Click this to copy the command and then fill
in the blanks!
apihost and apikey information is on the Service credentials tab.
Restart your Integration Node for the changes to take effect.
23
The Service Credentials section is pre-populated with an apikey which has been
created on your behalf
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
You use the apihost and apikey when activating the Bluemix reporting feature.
24
You can group your Integration Nodes and Integration Servers into
groups. For example: Dev, QA, Stage, Production
Usage data can be seen for a group of Integration Servers or an
individual Integration Server.
25
Choose between:CPU time, Number of active CPUsResident Set Size
Usage Details Advisor
View Integration Node details
View a list of recommended
services from the Bluemix catalog.
26
[sanjayn@sachin iib-10.0.0.8]$ mqsichangebluemixreporting LINUX_NODE -g -l active -r us-south -o [email protected] -a "Sanjay Nagchowdhury" -u [email protected] -p XXXXX -dRequest Bluemix logging tokens: https://logmet.ng.bluemix.net/login
Successful HTTPStatus: 200
Publishing Kibana dashboard
Successful HTTPStatus: 200
Response: Your Kibana 4 dashboards can be accessed at the following location: https://logmet.ng.bluemix.net/app/#/kibana4The Kibana 4 dashboards that have been uploaded include: IBM Integration Bus Dashboard
BIP8377I: Current dashboard has been loaded into Kibana for Bluemix logging space 'Sanjay Nagchowdhury'. Dashboard files are also prepared for manual import into Kibana from '/home/sanjayn/iibconfig/BluemixReporting/LINUX_NODE'.
BIP8071I: Successful command completion.
Reporting logging information to a Logmet service in IBM Bluemix and displaying it in a Kibana dashboard
Run mqsichangebluemixreporting to configure a connection to the Logmet service in Bluemix and send it logging information.
Go to this URL in a web browser to see logging infromation in a dashboard. Command can be repeated for multiple
Integration Nodes and Integration Servers to see logging information in a single dashboard.
NEW !
27
Example using Integration Nodes running on Linux and Windows.
Top 5 hosts that are generating log events
Top 5 integration servers that are
generating log events
Top 5 log events across the Integration Servers for a timeslot.
All events by severity.
Custom visual which shows Integration Servers that are stopping/starting/changing
28
This spike shows that a sudden increase in error messages occurred.
You can zoom in and see precise BIP messages that were generated in this
timeslot.
29
Chart after zooming in on the timeslot to check the errors.
Bottom table shows the logging messages that were recorded from the different
Integration Servers.
You can see errors that were generated.
You can see errors that were generated.
30
Effective Administration…
Installation
Configuration
Deployment
Migration
Monitoring
Security
31
Security
Admin security is optional, it is not enabled by default– Control user access to Integration Node resources– Enable through mqsichangebroker / mqsicreatebroker
Administration Security
Authentication Authorisation
“You are who you say you are!” “But what are you allowed to do ?”
Credentials checked by….
Web user account with a local password.Local user authenticated by OS
Integration Node LDAP ServerWeb user account with a password held in the LDAP Server
Permissions checked using…
Queue-based authorisation File-based authorisation
SYSTEM.BROKER.AUTHSYSTEM.BROKER.AUTH.EG
MQ INQ
MQ PUT
MQ SET
File Read
File Write
File Execute
33
SecuritySet mode to file-based
authorisation
Add web user account with a local password
Web UI now requires userid/password
User can view the Integration Server and its contents but
cannot start/stop/deploy
Set permission to view Integration Server, but not start/stop/deploy
34
Security
XXXXXX
Set the LDAP server that you want to use for authentication.
Configure Integration Node to connect to the LDAP Server.
Add a web user account for a user that is defined in the LDAP
Server.
Use a wildcard to add all users.
NEW !
35
Effective Administration…
Installation
Configuration
Deployment
Migration
Monitoring
Security
36
Supported Migration Paths
You can migrate to IBM Integration Bus Version 10.0 from the following previous versions:
• WebSphere® Message Broker Version 7.0.0.5
• WebSphere Message Broker Version 8.0
• IBM Integration Bus Version 9.0
Migration
Co-existence: IIB V10 can co-exist with IIB V9, WMB V8 and V7.
Windows : • Multiple instances of each of the versions (but different fixpacks) can be installed on the same system.
UNIX and z/OS : • Multiple instances of each of the versions and fixpacks can be installed on the same system.
Ensure correct mqsiprofile is sourced before starting the Integration nodes to pick up the correct versions.
Multiple versions of IIB Toolkit can be installed on the same machine • You can not use IIB V10 Toolkit to connect to a V9 broker or vice versa
39
MigrationMigration Options In-place migration:
• Migrates existing broker and its components immediately to V10
• Should be performed on the same system where broker exists
• Requires stopping the broker / integration node
• Requires running the command: mqsimigratecomponents • Provides –t option to roll back to previous state
Parallel migration:
• Provides ability to migrate on same or a different system Does not overlap with the existing broker, so no need to stop the broker
• Create new Integration node and deploy artifacts to it
• Requires the Integration node / server properties and Administration security to be reconfigured on the newly created components
1
2
3
4
Install V10
Stop Broker
mqsimigratecomponents
Start Integration Node
1
2
3
4
Install V10
Create new Integration Node
Deploy artefacts
Stop old Broker
Backup resources first!
40
Demo !
MongoDB
Switch Server
LINUX_NODE
WINDOWS_NODE
42
43
Effective Administration…
Installation
Configuration
Deployment
Migration
Monitoring
Security
44
IIB Sessions at Interconnect 2017Session Who Time
2110A What's New in IBM Integration Bus BT Monday 16:15 – 17:00
2141A IBM Integration Bus Futures and Strategy (Inner Circle only) BT Tuesday 11:30 – 12:15
2158A Technical Introduction to IBM Integration Bus GG Tuesday 13:30 – 14:15
2118A Developing Integrations for IBM Integration Bus on Cloud GG Tuesday 14:30 – 15:15
2144A IBM Integration Bus Customer Roundtable BT Tuesday 15:45 – 16:30
2121A Docker and IBM Integration Bus GG Wednesday 09:00 – 09:45
2151A Effective Administration of IBM Integration Bus SN Wednesday 10:15 – 11:00
2144B IBM Integration Bus Customer Roundtable BT Wednesday 16:15 – 17:00
2124A Operational and Business Monitoring with IBM Integration Bus SN Thursday 09:30 – 10:15
2111A IBM Integration Bus and REST APIs SN Thursday 10:30 – 11:15
2166 IBM Integration Bus Version 10 Hands-On Scheduled Lab GG+SN Monday 13:00 – 14:45
9402 IBM Integration Bus Version 10 Hands-On Open Lab None Any Open Lab Session
In case powerpoint isn’t your thing …
https://developer.ibm.com/integration Lots of Blog entries, regular updates and links to product demo
videos! All our recent enablement material is on youtube
IIB and Kibana dashboards https://youtu.be/sCPrT2dHKSs
Running IIB in Bluemix Container Service https://youtu.be/ybGOiPZO3sY
IIB and Kibana dashboards https://youtu.be/sCPrT2dHKSs
IIB and Hybrid Connect https://youtu.be/gWbxIooq3_g
IIB and LDAP https://youtu.be/HrqY9MyfzNs
IIB LoopBack Request node https://youtu.be/rUK_OQ5-Anw
Using IIB to integrate with MongoDB and Cloudant https://youtu.be/Is1pphngUlM
Using IIB for REST, Graphical Mapping & Salesforce: https://youtu.be/XIK6QvNSHdY
IIB, Kafka and Twilio SMS: https://youtu.be/7mCQ_cfGGtU
Using Kafka with IIB https://youtu.be/kYv0crxL86Y
Consuming REST APIs using the IIB REST Request node https://youtu.be/C_6gPlrCHZQ
Easy demo of an IIB App Connect node https://youtu.be/StwPbOiFKzk45
46
Notices and disclaimersCopyright © 2017 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.
U.S. Government Users Restricted Rights — use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. This document is distributed “as is” without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.
IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply.”
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented
as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.
47
Notices and disclaimers continuedInformation concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a particular, purpose.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services®,Global Technology Services®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli® Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
48
InterConnect2017
49
Back up
50
Effective Administration…
Topology Planning
Installation
Configuration
SecurityDeployment
Migration
Monitoring
High Availability
51
• What does your SLA say?• Agree SLAs with the business
(% uptime)• Scheduled and unscheduled
outages
• Ensure you have the technology to meet the SLA
• Redundancy in applications, • Clustering, • PowerHA (HACMP),• What about applying service?
Availability % Downtime per year Downtime per month Downtime per week
90 (one nine) 36.5 days 72 hours 16.8 hours
99 3.65 days 7.2 hours 1.68 hours
99.9 8.76 hours 43.2 min 10.10 min
99.99 52.6 min 4.32 min 1.01 min
99.999 (“five nines”) 5.26 min 25.9 secs 6.05 secs
99.9999 31.5 secs 2.59 secs 0.605 secs
A CB1L1 L2
B2
A CBL1 L2 89.996%
Availability[~2.4 hours down per day]
98.996%Availability[~14.5 mins down per day]
Example• Same processing components: A, L1, B, L2, C
• Same levels of availability for each component• A, C, L1, L2 are 99.999% available. • B is 90% available
• So duplicate B B1 and B2• Individually B1 and B2 are 90% available, but
together they are 99% available.
• Increased availability by duplicating components
• How do I ensure that the integration node is continually processing messages?• Active/Active vs. Active/Passive
52
High Availability Each Integration
Node operates independently
Continuous availability of the service during a failure
Single instance of IIB
Active & Passive state must have identical copies of persisted data.
53
High Availability
Multi-instance queue manager• Integrated into the IIB and MQ products• Faster failover than HA cluster• Delay before queue manager restart is much shorter• Runtime performance of networked storage must be considered• IP address of standby instance is different to primary• No automatic fail-back to primary hardware when restored
HA cluster• Capable of handling a wider range of failures• Failover historically rather slow, but some HA clusters are improving• Some customers frustrated by unnecessary failovers• Extra product purchase and skills required
IP address of each machine is different.
Queue Manager restart quicker.
Single IP address.
Handle wider range of failures.
54
High Availability
Planning for disaster recovery• What would you do if your primary IIB location goes down?
• Distribute IIB to multiple sites if possible• This introduces data replication and latency concerns
• Keep DR concerns separate from HA!
Disaster Recovery High AvailabilitySystems at multiple sites with replicated configurations
Systems at a single site with a single configuration
DR is unplanned. An HA failover can be a planned activity.
Take regular backups!
Write a DR plan!
Test your DR plan!
55
Effective Administration…
Topology Planning
Installation
Configuration
SecurityDeployment
Migration
Monitoring
High Availability
QM1
IntegrationNode 1
QM2
IntegrationNode 2
sprayer
MQ requests
Machine A Machine B
Integration Node 1 has flows getting/putting to QM1 and QM2
Integration Node 2 also has flows getting/putting to QM1 and QM2
Topology Planning
QM1
IntegrationNode 1
QM2
IntegrationNode 2
sprayer
MQ requests
Machine A Machine B
If QM1 goes down, Integration Node 1 and Integration Node 2 are unaffected. They can continue to get/put from QM2
QM1
IntegrationNode 1
QM2
IntegrationNode 2
sprayer
MQ requests
Machine A Machine B
If Integration Node 1 goes down, but QM1 is still running, then Integration Node 2 is unaffected. It can continue to get/put from QM1 and QM2
Topology Planning
NEW !
Node wide HTTP listener
SYSTEM.BROKER.WS.INPUT
http://localhost:7080 https://localhost:7083
http://localhost:7080/ServiceC
http://localhost:7080/ServiceD
HTTPHTTPS
SYSTEM.BROKER.WS.REPLY
http://localhost:7080/ServiceA
http://localhost:7080/ServiceB
Integration Server
Integration Server
Integration Node wide HTTP(S) Listener
biphttplistener
Node wide HTTP listener
SYSTEM.BROKER.WS.INPUT
http://localhost:7080 https://localhost:7083
http://localhost:7080/ServiceC
http://localhost:7080/ServiceD
SYSTEM.BROKER.WS.REPLY
http://localhost:7080/ServiceA
http://localhost:7080/ServiceB
Integration Server
Integration Server
Integration Node wide HTTP(S) Listener
Servlet Container
Proxy ServletHTTPHTTPS
biphttplistener
60
Node wide HTTP listener
SYSTEM.BROKER.WS.INPUT
http://localhost:7080 https://localhost:7083
http://localhost:7080/ServiceC
http://localhost:7080/ServiceD
SYSTEM.BROKER.WS.REPLY
http://localhost:7080/ServiceA
http://localhost:7080/ServiceB
http://localhost:7800/ServiceA
Threads
HTTP(S) Listener
http://localhost:7801/ServiceC
Threads
HTTP(S) Listener
Integration Server
Integration Server
http://localhost:7800/ServiceB
http://localhost:7801/ServiceD
HTTPHTTPS
HTTPHTTPS
Integration Server
Integration Server
Integration Node wide HTTP(S) Listener
Integration Server HTTP(S) Listener
Servlet Container
Proxy ServletHTTPHTTPS
biphttplistener
61
Node wide HTTP listener
SYSTEM.BROKER.WS.INPUT
http://localhost:7080 https://localhost:7083
http://localhost:7080/ServiceC
http://localhost:7080/ServiceD
SYSTEM.BROKER.WS.REPLY
http://localhost:7080/ServiceA
http://localhost:7080/ServiceB
http://localhost:7800/ServiceA
Threads
HTTP(S) Listener
http://localhost:7801/ServiceC
Threads
HTTP(S) Listener
Integration Server
Integration Server
http://localhost:7800/ServiceB
http://localhost:7801/ServiceD
HTTPHTTPS
HTTPHTTPS
Integration Server
Integration Server
Integration Node wide HTTP(S) Listener
Integration Server HTTP(S) Listener
Servlet Container
Proxy ServletHTTPHTTPS
biphttplistener
Export port configurations and use
in an external HTTP Server
Export port configurations and use
in an external HTTP Server
62
Effective Administration…
• Most resilient and best performing systems are those that are:• Loosely coupled and have parallel execution (threads and processes)• Implications for your message flow design will depend on the systems that IIB interacts with
• Vital to conduct performance testing before production in production like environment• Gives time to evaluate and refactor code if needed
• Top Tips• Always ensure your environment is reproducible• Treat DR and HA separate• Ensure regular backups• Encourage developers to create message flows that enables operational tweaks to be made• Schedule regular maintenance windows
Summary
63