case merging ibm mq-based products with other study ... · y ibm datapower y ibm integration bus...

© 2017 Nastel Technologies, Inc. • 48 South Service Road • Melville, New York, NY 11747 • 1-631-761-9100 • nastel.com

Merging IBM MQ-based Products with Other Messaging Solutions

C a s eStudy

© 2017 Nastel Technologies, Inc. • 48 South Service Road • Melville, New York, NY 11747 • 1-631-761-9100 • nastel.com 2

In the Beginning, There Was Darkness...

Ok, I’m being a bit overly dramatic. What was happening in New York a couple of years ago at a really big global bank with almost U.S. $2 trillion in total assets—let’s call it MegaBank—was a serious lack of control.

Let me paint the scene. The IT team was getting along pretty well... Among other development and oper-ations activities, they were successfully monitoring and managing their IBM MQ (formerly known as WebSphere MQ) middleware family of products as well as a homegrown messaging solution. All was well with bank applications utilizing IBM MQ and its associated tools.

So what was the problem? MegaBank acquired another firm—which I’ll dub TargetBank—that employed TIBCO EMS (a fully compliant JMS service implementation) and TIBCO Rendezvous (messaging middleware) instead of IBM MQ for their middleware and messaging layer. The process of combining two

completely different infrastructures was creating a major disturbance in The Force.

But being experienced pros, MegaBank’s IT team quickly set about putting DataPower appliances to work acting as a translation bridge between the two technology platforms, transforming and routing messages coming from MegaBank applications using IBM MQ into the native TIBCO EMS format that TargetBank’s applications employed, and vice versa.

To monitor and manage performance of the combined middleware and messaging infrastructure end-to-end, the team needed visibility for:

y Their custom applications y IBM DataPower y IBM Integration Bus (IIB) y Solace Message and Content Routers y IBM MQ y TIBCO EMS and TIBCO Rendezvous (RV)

...and they wanted a single product to handle it all.

About the Author

Charley Rich has decades of product management experience spanning Middleware Messaging, Analytics, APM, and SaaS. Charley contributed to four successful startups and was worldwide product manager at IBM Tivoli, earning the General Manager’s Award. At Nastel Technologies he is VP – Product Management, and as a published author and sought-after industry speaker brings light to the Dark Side of technology environments whenever possible.


Unfortunately, MegaBank’s home-grown tools couldn’t handle the complexity of the new tech-nology environment. Something better was needed to help maintain improved productivity, reduce MTTR (mean time to repair) for problems, and keep costs associated with problem resolution at acceptable levels.The search was on for a global monitoring solu-tion that could provide a single point of control for:

y Unified operational procedures – that span mul-tiple technology platforms

y Run-time operations – encompassing IBM MQ administration, rapid problem detection and isola-tion, and operational review and planning

y Message auditing – used to trace the granting or denying of permissions

y Tracking – stitching together a flow of transac-tions, each defined as a group of activities targeted at achieving a common goal or task. These activ-ities are deemed related based on observation of messages exchanged between them or more precisely between logical units of work (LUWs). LUWs represent a collection of operations and messages within a session that must all be done or none done, and are typically delimited by START/COMMIT calls

y Transformation – used in IBM Message Broker (IBM Integration Bus) where the structure of a mes-

Figure 1: MegaBank required monitoring, tracking, and analytics for their complex infrastructure


sage is changed from one XML format to another. This is often required to provide communications between applications that read different formats

y Routing – employed in IBM Integration Bus (Mes-sage Broker) to send a message to a receiver based on message contents

y Performance – measuring application throughput in terms of metrics such as messages/second or transactions/second

The right tool would help improve interoperability between IBM DataPower, Solace, IBM MQ and TIBCO. Last but certainly not least on the team’s wish list was the ability to track messages through JMS, IBM MQ and DataPower.

Would they be able to find a single software product that could do everything and save their universe?

Of course they did!

Before MegaBank’s merger with TargetBank, their IT department had been using a well-known event monitoring product to monitor IBM MQ. TargetBank used TIBCO Hawk to monitor their TIBCO EMS system.As the bank merger progressed, business require-ments soon dictated that a seamless integration across the combined environment was necessary. But at this point, they had two separate management systems. MegaBank needed a tool that could handle their home-grown message broker plus multi-vendor middleware and messaging environments.

After due diligence, running a Proof of Concept (POC), and evaluating deployment, operations and ability to satisfy key use cases, the IT team chose Nastel’s AutoPilot® product to monitor and manage IBM MQ on their distributed and mainframe z/OS platforms.

Why? One compelling reason was that AutoPilot’s array of fully integrated middleware monitoring, tracking and administration capabilities could super-vise the bank’s homegrown message broker along with its internal business applications.

(Now before you the reader think I’m just making a naked sales pitch for my company, I want to say there are many fine products on the market. But in this particular instance Nastel turned out to be a particu-larly good fit for MegaBank’s business challenges, as described later.)

As MegaBank moved ahead with integrating its messaging systems, the decision was made to do likewise with their middleware monitoring systems. As a result, the IT team charged with integrating front-end and back-end applications also absorbed all middleware-related responsibilities.

They soon encountered additional issues as the two IT infrastructures began to merge. Mission-critical bank applications that relied on IBM and TIBCO products—including a particularly critical high-volume trading application—began to falter under a full production workload consisting of millions of messages per second over hundreds of queue managers.

With multiple monitoring systems assigned to the IBM and TIBCO products, there was no clear consensus on the root cause of a problem as it occurred, or even if there was an actual problem!

MegaBank’s IT specialists were trained on either the IBM product or the TIBCO, but few on both, making collaboration challenging. Complicating things further was the fact that the incumbent monitoring


solutions were producing far too many false alarms. The support team felt like they were chasing ghosts.

Another concern was the lack of integrated moni-toring for DataPower and Solace appliances. Having to juggle multiple monitoring solutions along with possible needs for a third or fourth would prevent the IT team from meeting performance standards. So they determined a single monitoring solution was necessary that would:

y Find stuck messages y Deliver early notification of faults, bottlenecks and

slowdowns y Monitor the performance of messaging and com-

parisons to SLA performance requirements y Supply administrative control over IBM MQ y Furnish complete message tracking through JMS,

IBM MQ , and DataPower y Easily integrate with the bank’s enterprise manage-

ment systems (EMSs) y Provide complete real-time visibility across Dat-

aPower, Solace, TIBCO, IBM MQ and MegaBank’s home-grown middleware solution

Relief was on the way for the IT team. Nastel support specialists went on-site with MegaBank, learned their unique requirements, and devised an effective archi-tecture for them.

Installation was accomplished rapidly with careful attention directed at devising sophisticated policies (sets of event processing rules) to handle specific use cases automatically as “situations.” This avoided a simple thresholding approach, which would have resulted in numerous false alarms.

Bringing Light to the Dark Side…

Nastel’s AutoPilot was already effectively monitoring MegaBank’s IBM MQ and homegrown MOM (manager of managers) systems. However, while the bank trusted TIBCO, the IT team wanted to integrate TIBCO messaging products into an umbrella monitoring framework.

But was this a “nice-to-have” or a “must-have” situ-ation? It proved to be the latter, because in large enterprise environments like MegaBank’s, with many interdependent applications constantly in flux, the speed and ease of adapting to constant change is key. Although AutoPilot enables the addition of new EMS servers or Rendezvous daemons in minutes, config-uring TIBCO Hawk to monitor them wasn’t so much a problem of time; instead, it was a silo approach to the general need for middleware availability.

The best technique would be to consolidate to fewer, or ideally, a a single monitoring system. Better still would be to implement appropriate policies and decrease old-school “eyes-on-screen monitoring,” with a focus on automating problem detection and optimizing false alarm suppression.

Bearing in mind the bank’s desire to consolidate software systems where possible, and ever-present IT resource constraints, deploying AutoPilot monitoring was an easy choice. As an added benefit and cost saving, IT team member training would be limited to a single monitoring system.

So the IT team was making progress. By standard-izing all application performance monitoring under


Nastel AutoPilot, MegaBank gained significantly by simplifying procedures, reducing vendor counts, and improving the overall quality and efficiency of moni-toring. Specifically, the team would benefit from:

y Unified monitoring and management procedures across all middleware technologies with a single tool

y A single portal for administration and certification of user permissions and access

y Easy and fast creation of powerful processing rules (utilizing Nastel’s Complex Events Processing engine) that could be applied to all middleware transports for end-to-end monitoring

Taking the Pulse of Solace

AutoPilot would monitor multiple Solace appliances, harvesting their events and analyzing them in correlation with other middleware and system events via the AutoPilot CEP (Complex Event Processing) engine. By combining monitoring of Solace software environment components with hardware inventory and assets (e.g., blades, power modules, fans, etc.),

the IT team arrived at a comprehensive command and control solution.

Analytics included:

y Appliance specifications y Appliance incoming/outgoing data statistics y Cache and cache-level statistics y Router clients and queues y Hardware properties and specifications y VPNs and VPN-level statistics y Current/max messages spooled

Measuring DataPower

The deployment of IBM DataPower appliances for message transformation and routing between the IBM MQ and TIBCO middleware messaging systems played to Nastel’s strengths in both middleware monitoring and message tracking. The ability to provide deep operational monitoring for DataPower and provide a unified view across all middleware messaging solutions was important to MegaBank’s IT team.

Figure 2: DataPower transaction topology


Nastel AutoPilot for DataPower provided analytics on:

y System resource consumption over time y Metrics y Performance of key business activities y Routing and transformation history y Correlation of DataPower performance data with

events from other systems

With unified monitoring and control capabilities established for the messaging appliances deployed across the combined infrastructure landscape, MegaBank’s team was ready to complete the task of uniting all middleware monitoring under the AutoPilot umbrella.

A major driver of MegaBank’s decision to standardize on Nastel AutoPilot was its ability to monitor all of its middleware systems. With AutoPilot’s TIBCO EMS plug-in, the IT team could easily monitor TIBCO EMS servers and components including: queues, topics, consumers, producers from a single, consolidated vantage point.

The Backstory on MegaBank’s TIBCO EMS Monitoring Solution

At MegaBank (or any other customer utilizing TIBCO), the TIBCO EMS (Enterprise Message Server) Plug-in Expert (a “plug-in expert” is a collection of polices and instrumentation to automate the monitoring of a specific technology, e.g., JMX) is modularized based on EMS servers. The AutoPilot TIBCO EMS Plug-in retrieves information such as queues, topics, consumers, producers and more, plus their associated components. For example: queue expert, topic expert, consumer and producer experts, etc.

A customer’s specific requirements for performance monitoring activities determine how plug-in experts are utilized—individually or in any combination of multiple deployments. AutoPilot’s TIBCO EMS Plug-in Expert automatically measures and collects EMS performance metrics, periodically gathering informa-tion about TIBCO EMS servers and their components on a sampling interval defined by an IT middleware team. Some key performance metrics collected include:

y Inbound y Outbound y Acknowledged and pending message volumes y Message rates per second y Average response times y Message tracking y Age of oldest messages

These metrics are collected for servers, queues, routes, groups, consumers, producers, transports, and durables. (Creating a “durable” data subscription means it is stored on the server and subscribers can receive messages even if it was inactive when the message was originally delivered. Otherwise, if messages are published when the subscriber is not available, the subscriber does not receive those messages and events.)

AutoPilot’s powerful Complex Event Processing (CEP) engine uses these metrics in policies an IT middleware team defines for handling predictions, correlations, validation, automation, notifications, logging and alerts. It routinely combines metrics and data from many diverse systems within a customer’s infrastructure.


In the case of MegaBank, one of the most important features for the firm was the fact that the AutoPilot TIBCO/EMS plug-in easily monitors multiple TIBCO EMS servers. (Actually, there’s no limit to the number of TIBCO EMS servers that could be simultaneously monitored.)

The AutoPilot CEP engine correlates EMS metrics with measurements from other infrastructure components such as DataPower, Solace, IBM MQ, Message Broker, ESB, and other supported technologies, dynamically creating events and alerts for IT staff as needed.

Lifting the Hood on MegaBank’s TIBCO Rendezvous Monitoring Solution

An additional requirement MegaBank had for AutoPilot was performance monitoring for TIBCO Rendezvous (RV) and visibility into TIBCO multicast services. Using AutoPilot’s TIBCO RV Plug-in expert, the IT middleware team monitored TIBCO RV performance statistics.

The plug-in automatically collected data on the TIBCO RV server and its components (services, routers and transports), gathering TIBCO RV statistics by subscribing to and processing its advisory messages. The collected data was analyzed and provided performance analytics for:

y TIBCO RV Server Info y Client transport y Service y Router info y Security info

MegaBank needed this functionality in both agent and agent-free deployments. (In some environments,

the bank was unwilling to deploy anything additional to their production servers and therefore required an agent-free implementation). The TIBCO RV Plug-in utilized metrics provided by the TIBCO RV server; this data is used in AutoPilot business views for validation, automation, notification, logging and alerts.

An AutoPilot Business View showing the health of the TIBCO RV Server environment displayed RV status, including:

y Server health state y Inbound and outbound multicast lost messages y Inbound and outbound point-to-point lost

messages y Number of clients using a particular service y Number of hosts using a particular service y Rate of inbound and outbound traffic y Amount of inbound and outbound data y Number of missed inbound messages

AutoPilot provided MegaBank’s IT team with the visi-bility and performance metrics they needed to prop-erly monitor TIBCO messaging components—just as it did for their IBM MQ-based family of products.

Messaging Middleware and the Importance of Message Tracking

Regardless of vendor, today’s messaging middleware has transformed the way applications communicate and exchange information. Application integration has brought about significant advantages to the development of composite applications. However, it has also brought about a need to track messages end-to-end through JMS, IBM MQ, DataPower, message brokers and other middleware.


As IT complexity has grown, compliance and inte-gration needs have evolved, creating a critical need to track underlying message exchanges between IT services and their corresponding business services. IT professionals who support critical business applica-tions must deal with the problematic consequences of performance degradation or messages that fail to make it to their respective destinations. When messaging failures occur, they can be quite difficult to resolve and take many hours to troubleshoot.

Real-time message tracking provides critical visibility into message flows, enabling organizations to prior-itize problems appropriately and work on issues that have the most impact to the business, such as:

y Where is my message? y Where is my order? y Are orders flowing as expected?

Many problems turn out to be false alerts. The sophis-ticated rule-based processing capabilities available in AutoPilot’s CEP (Complex Event Processing) engine can suppress most if not all of these false alerts while simultaneously providing IT teams with information they’re keenly interested in: pro-active notification about pending disruptions in message flows and stuck or hung messages.

Benefits from Message Tracking Capability

MegaBank’s middleware team now had the trouble-shooting capabilities they needed to quickly resolve issues their newer applications were facing in a more complex, combined IT infrastructure. With AutoPilot’s built-in message tracking facility, the team was able to extend the unified “Big Picture” management

Figure 3: MegaBank’s transaction tracking, analytics, notifications, and alerts


approach they had pursued for general middleware monitoring and administration.

AutoPilot provided the following benefits:

y Real-time tracking of JMS, IBM MQ and DataPower middleware message flows (transactions)

y Instant detection of delayed, compromised or incomplete JMS, IBM MQ and DataPower middle-ware message flows (transactions)

y “Dope and trace” and “recognize and trace” tech-nologies in conjunction with unique transaction IDs enabled stitching and inferencing of the mes-sage stream

y Notification and automated actions based on us-er-defined criteria

y Middleware payload data could be used to stitch messages together based on business policy

y IBM MQ transaction throughput and message vol-ume analysis performed in real-time

y Storage of all performance-related information in an easily accessible SQL database

y Audit trail for all transactions and messages y IBM MQ transaction activity reports on a daily,

weekly, monthly basis y Policy-driven monitoring and automated

notification y Identification of the root cause of performance

bottlenecks

I’ve talked about some particulars associated with one large financial institution, MegaBank. But what can people expect from AutoPilot in general with respect to performance and scalability issues? Let’s examine some topics…

Horizontal Scalability

y On the agent level: AutoPilot can monitor a large number of applications across thousands of servers

Figure 4: MegaBank consolidated dashboard showing a datacenter, server, geo, and application view


y On the server level: AutoPilot’s Complex Event Processing (CEP) engine can span multiple servers utilizing its built-in grid

y On the user level: AutoPilot can handle thousands of users

Vertical Scalability

AutoPilot can process large volumes of data from one source (one server that may have multiple applications). For example: situations where one queue manager had more than 20K queues and 10K channels, which translated to hundreds of thousands of data metrics.

Transactional Metrics

The performance impact on applications tested with JSpec resulted in an impact of less than 2.5%.

Input Speed

AutoPilot has the capability to process consistently high volumes of KPIs (key performance indicators) and tens of thousands of transactions per second, per CEP instance. Because CEP engines can be added on-demand, processing capacity can be increased elastically as needed.

Rules Processing Speed

A single AutoPilot CEP engine running on a 64-bit quad-CPU server with 4 GB of memory can process 2M rules per second. Because the CEP engine is a virtual machine it scales linearly, so adding an addi-tional CEP engine can double throughput.

High Availability

AutoPilot’s grid-based infrastructure provides for a smooth failover. It supports monitoring for a wide range of messaging middleware platforms including: IBM DataPower, IBM MQ , IBM ESB, IIB (Message Broker), WBI, TIBCO RV and EMS, Solace Systems Messaging Appliances, CICS and JMS.

Conclusion: AutoPilot Is a Powerful Monitoring Solution for Large Enterprises

In the case of MegaBank (and many other large enterprises around the globe), AutoPilot has proven a highly effective, comprehensive and pro-active solution for the bank’s complex, mixed middleware environment. It is the only monitoring solution that supports traditional software-based messaging technology as well as appliance-based technology, enabling the same operating procedures to be utilized for all messaging products. Since standard-izing on AutoPilot, the IT middleware team has real-time, end-to-end visibility and easily pinpoints problem areas—and their root causes—anywhere across its various middleware systems.

AutoPilot’s Complex Event Processing (CEP) engine automatically eliminates false alerts, identifies potential problems, and provides early warning and prediction of infrastructure problems. The net result is avoidance of business process disruptions that adversely impact end-users and customer experiences. AutoPilot’s CEP capabilities dramatically reduce the mean time to problem resolution, increase overall uptime, and minimize user support


requirements by cutting down significantly on support calls from business users to the IT middleware team. AutoPilot is the only monitoring solution that unifies user “permissioning” and certification of access across the middleware estate to ensure regulatory compliance is met.

Real-time visibility on business processes obtained from AutoPilot’s message tracking facilities lets orga-nizations not only measure how well their business services perform at any given moment, but optimize them end-to-end.

Perhaps the most striking outcome is that AutoPilot enhances an organization’s responsiveness by enabling its IT team to focus on the right issue, at the right time and place. This is clearly demonstrated in the example of messages being tracked as they span various nodes in the network structure of an orga-nization’s infrastructure—with performance bottle-necks pinpointed whenever they occur, and most important: why. This lets IT teams focus on problem resolution as issues arise over time with both speed and precision.

AutoPilot is the basis for monitoring, automation, visualization, and fault recovery for all of MegaBank’s multi-vendor middleware, messaging appliances, and brokers—end-to-end. The IT middleware team keeps their entire infrastructure governed, visible, and manageable from a single pane of glass, and enjoys the additional benefits of reducing the number of

software vendors, plus reducing training expenses, and overall IT human resource demands.

About Nastel AutoPilot

Nastel Technologies helps the world’s largest enter-prises solve application compliance, surveillance and performance issues while safeguarding customer relationships and lowering operational costs. Nastel’s Enterprise-Grade APM solution encompasses deep real-time monitoring, transaction tracking, and analytics spanning applications, middleware, transac-tions, end-user experience, logs, and mobile services in a single, powerful, easy-to-use solution.

About Nastel Technologies

Nastel Technologies helps the world’s largest enterprises solve application compliance, surveillance and performance issues while safeguarding customer relationships and lowering operational costs. Nastel’s enterprise-grade solutions encompass deep real-time monitoring, transaction tracking, and analytics spanning applications, middleware, transactions, end-user experience, logs, and mobile services in a single, powerful, easy-to-use solution.

Nastel is privately held and headquartered in New York, with offices in the U.S., the U.K., France, Germany and Mexico, with an additional network of partners throughout Europe, the Middle East, Latin America and Asia. For more information, visit nastel.com. n

case merging ibm mq-based products with other study ... · y ibm datapower y ibm integration bus...

Documents