default messaging apars - ibm application server for z/os ... and forwarding function very similar...

WebSphere Application Server for z/OS

Default Messaging APARsThe results of an IBM review of customer PMRs related to the

Default Messaging and SIBus function of WAS z/OS

Version Date: May 23, 2014

See "Document Change History" on page 13 for a descriptionof the changes in this version of the document

IBM Advanced Technical SkillsGaithersburg, MD

WP102450 atibm.com/support/techdocs

© IBM Corporation 2014

WAS z/OS Default Messaging and SIBus Function APARs

Many people were involved at one point or another in the analysisand resolution of issues addressed by the APARs:

IBM US: David Bonaccorsi, David Follis, Thomas Henry, DavidHu, James Mulvey, Gary Picher, Michael Poirier, Nathan Rauh,

Frederick Rowe, Michael Stephen

IBM UK: Russ Heald

IBM India: Balram Giridhar, Nihilson Gnanadason, SanjayKesavan, Nitik Kumar, Padmakumar Nambiar, Lalitha Pannala,

Sonali Samantaray, Urwashi Sharma, Lohitashwa Thyagaraj

© 2014, IBM CorporationAmericas Advanced Technical Skills - 2 -

WP102450 at ibm.com/support/techdocsVersion Date: Friday, May 23, 2014


Table of Contents

Executive Overview...............................................................................................................................4Two areas of focus................................................................................................................................4APAR Summary....................................................................................................................................4

Focus Area 1 – Server Startup in Recovery Mode..............................................................................5Overview of “recovery mode”................................................................................................................5Overview of issues that resulted in customer PMRs..............................................................................5Corrective APARs.................................................................................................................................5Summary...............................................................................................................................................6

Focus Area 2 – Availability of Messaging Engine Function...............................................................7Overview...............................................................................................................................................7Important notes about context...............................................................................................................7Corrective APARs.................................................................................................................................8Additional control functions with PM76557............................................................................................8General Recommendations for new settings.......................................................................................11

Document Change History..................................................................................................................13




Executive OverviewIBM's WebSphere Application Server1 includes a messaging function known as "Default Messaging," or the "Service Integration Bus." This Default Messaging provides a message storage and forwarding function very similar in concept to IBM's WebSphere MQ product2.

On WAS z/OS the Default Messaging function is implemented in a separate server address space from the Controller and Servant. That separate address space is called the "Control Region Adjunct," or CRA for short:

The CRA is where the code runs that implements the "Messaging Engine" (ME) function3. It is the ME that implements the messaging destinations (queues) and manages message persistence to file or relational data store. Within an application server cluster each member of the cluster will have a CRA, with the Messaging Engine function operating with a failover and failback mechanism within the cluster4.

This document is a result of development and testing work performed within IBM in response to several customer PMRs5 related to the Default Messaging function. This document will serve as a vehicle to communicate information about the APARs6 taken from the PMR investigation.

Two areas of focus

The analysis and testing focused around two areas of functionality:

1. Recovery Mode startup for servers configured with a Messaging Engine and CRA

2. The timing of availability of the Messaging Engine and Schedulers during server startup

APAR Summary

The following table shows the fixpack by version in which each APAR appears7:

Focus Area APAR Version 7 Version 8 Version 8.5 Details

1 PM68247PM87430PM89729PM98412

7.0.0.277.0.0.317.0.0.337.0.0.33

8.0.0.68.0.0.88.0.0.98.0.0.9

8.5.0.28.5.5.28.5.5.28.5.5.2

Page 5

2 PM76557PM92447

7.0.0.317.0.0.33

8.0.0.98.0.0.9

8.5.5.28.5.5.2

Page 7

1 All platforms supported by WebSphere Application Server. This document focuses on WAS for z/OS.2 In fact MQ may be used in combination with the Default Messaging function. The two are not mutually exclusive.3 It's possible that a server's CRA will host more than one Messaging Engine, depending on how many SIBuses the server is attached to.4 The failover and failback behavior described here is associated with the "High Availability" configuration policy when the cluster is

assigned to the Service Integration Bus. An alternative is the "Scalability" policy, which has multiple active MEs with queue destinations partitioned between them.

5 Problem Management Record -- the mechanism used to alert IBM of potential defects with products.6 Authorized Program Analysis Report -- a formal recognition of a problem with an associated workaround or corrective fix.7 Or is targeted to appear, according to the text of the APAR.




Focus Area 1 – Server Startup in Recovery ModeOverview of “recovery mode”

"Recovery mode8" is a mechanism by which an application server may be started to recover transactions but not accept any new work during the transaction recovery process9.

When a server is started in recovery mode, the controller and servants start (and CRA, if present), initialize to a point where the transaction manager function can recover any transactions, then the controller, servants (and CRA) stop automatically10. No new work is accepted during this process.

With the transactions recovered the server may then be started normally at a time of your choosing.

WAS z/OS provides two ways to start a server in recovery mode: the first involves modifying thecontroller JCL start procedure, and the second involves adding a parameter to the MVS START command. Both result in the exact same thing.

Modifying the controller JCL involves changing REC=N on the JCL PROC statement to REC=Y:

//MYACRC PROC ENV=,PARMS=' ',REC=Y,AMODE=00 // SET ROOT='/wasv7config/mycell/mynodea' // SET FOUT='properties/service/logs/applyPTF.out'// SET WSDIR='AppServer'

By default that setting will be REC=N. Starting a server in recovery mode involves modifying the JCL, starting the server in recovery, then modifying the JCL back when complete.

The alternative is to issue the server START command with the REC=Y parameter specified:

S MYACRA,JOBNAME=MYSR01A,ENV=MYCELL.MYNODEA.MYSR01A,REC=Y

Note: When the server is a member of a cluster, one additional step is required. In the general properties for the cluster, see if the "Enable failover of transaction log recovery" checkbox is checked. If it is, then uncheck the box11 and start the server in recovery mode. When you restartthe server normally, re-check the box.

Overview of issues that resulted in customer PMRs

It was discovered that when a server is configured with a Messaging Engine (which means a CRA will start for the server), the CRA would not start in recovery mode. In-doubt transactions involving the server's Messaging Engine would not be recovered.

Note: This is related to a server started in recovery mode. A server started normally performs transaction recovery as well. The difference is in recovery mode the server accepts no new work. There was no issue with servers started normally. The issue involved servers with a configured Messaging Engine started in recovery mode.

In resolving that issue several other things came up. Each were addressed with a separate APAR.

Corrective APARs

The following table outlines the APARs created, a brief description of each, and information on fixpacks by version when the APAR became available.

APAR Description Fixpack by Version

PM68247 From the APAR: When starting the WebSphere Application Server z/OS Server in 'recovery mode' the Adjunct Region is not started.

This is the original issue with CRA not starting in recovery mode.

7.0.0.278.0.0.68.5.0.2

8 See the WAS Information Center and search on the unique string trun_svr_recoverymode9 During normal server start recovery is performed as well. The difference with “recovery mode” is that no new work is accepted.10 It may take a minute or two, but eventually the controller and CRA will stop. 11 The default is unchecked, but in your environment it may be checked to perform failover of the transaction log within the cluster.





PM87430 From the APAR: When application server is restarted in recovery mode, it may fail with errors if localhost on the system resolves to a different hostname than what the application server is expected to use.

Only an issue when the server's configured host name differs from what 'localhost' resolves to on that system.

7.0.0.318.0.0.88.5.5.2

PM89729 From the APAR: A ClassNotFoundException is thrown when loading com.ibm.ws.wmqra.zdispatcher.WMQRAMessageReferenceToken class. This happens when CRA dispatches work to SR on z/OS.

Corrects an issue created by the lack of the class file on the export list of the manifest file in the com.ibm.ws.sib.server.jar file.

7.0.0.338.0.0.98.5.5.2

PM98412 From the APAR: Message “WMSG0902E is issued when starting WebSphere Application Server z/OS in 'recovery' mode.” This message does not cause any problem, but should not be issued.

Corrects an issue with an error message being issued incorrectly. The issuance of the error message does not itself cause any problems, but the inclusion of the error message in the server output is incorrect and is fixed in this APAR.

7.0.0.338.0.0.98.5.0.2

Summary

The original issue was confined to a narrow use-case: servers with a configured Messaging Engine started in recovery mode. That issue was addressed with APAR PM68247. A few downstream issues resulted and those were addressed with APARs PM87430, PM89729 and PM98412.




Focus Area 2 – Availability of Messaging Engine FunctionOverview

As illustrated earlier, a WAS z/OS application server with configured Messaging Engines will consist of at least12 three z/OS address spaces: a Controller Region (CR), a Servant Region (SR), and a Control Region Adjunct (CRA). The CR is started first, and after that initializes to a point WLM starts the SR and the CRA. Eventually all three initialize fully.

There were two issues13 that needed to be addressed, both having to do with timing:

• An application server may contain a scheduler, which is a software function that initiates work to be performed in the server. The IBM Business Process Manager (BPM) productmakes use of schedulers as part of its operation. The timing issue here was the possibility of a scheduler coming active and initiating work before the application server was fully initialized; that is, all components ready to do work.

To address this an additional control property was added to indicate that a scheduler was not to begin work until the server as a whole was “open for e-business.” Details on that additional configuration value are provided on page 8.

• An application server with a configured Messaging Engine may “open for e-business” before the ME was fully active. That may create a situation where applications in the server would begin work expecting the ME to be active, but in fact the ME wasn't yet ready.

To address this the concept of a critical ME was introduced. A critical ME is a Messaging Engine that must be fully active before the server is allowed to be marked “open for e-business.”Additional configuration values are provided with APAR PM76557 to control the behavior of MEsmarked as “critical.” Details on those additional configuration values are provided on page 9.

The analysis into these two timing issues resulted in the creation of APARs PM76557 and PM92447. Details for these APARs are provided next.

Important notes about context

The timing issues outlined above are related to a fairly specific environment – that is, servers that have schedulers, or servers that are connected to a Service Integration Bus and host a Messaging Engine that uses “Data store” as the message store type14. Specifically, the following scenarios do not apply:

• An application server that uses IBM MQ as its JMS messaging provider is not affected by the timing issue. MQ provides the messaging engine function so there is no SIBus ME.

• An application server with a Message Driven Bean (MDB) application that uses JMS Activation Specs (as opposed to Listener Ports) listening on an MQ queue. In that case there will be a CRA, but without an SIBus there's no Messaging Engine. Therefore, the ME startup timing issue does not apply.

The use-case scenario of particular note is when IBM Business Process Manager (BPM) is involved. BPM makes extensive use of the SIBus as well as schedulers, and thus both the ME timing issue and the scheduler timing issue come up. APAR PM76557 came about in part because of BPM users who experienced issues with ME and scheduler startup timing.

The bottom line is that this APAR may not apply to your environment. It depends on whether you're using the SIBus and servers with Messaging Engines, or have servers with schedulers that are initiating work too early.

12 An application server may have 1 or more servant regions. But always just one CR and, if configured, just one CRA.13 The two are not directly related to one another. However, they are similar in concept, and products such as IBM Business Process

Manager make use of both schedulers and the Messaging Engines. Therefore, the corrective actions are found in the same APAR.14 “Data store” implies a database; “Filestore” implies message persistence in a file system location.




Corrective APARs

Two APARs were created:


PM76557 From the APAR15:

Messaging Engine should not start until after the 'Server Start Message' is issued. Individual Servant Region Address Spaces can be started and available prior to the full server being initialized. The start of the Messaging Engine prior to the Server being fully initializedcan cause problems.

As noted, this APAR provides not merely a corrective fix to the issue described, but additional control functions as well. Those are describednext.

7.0.0.318.0.0.98.5.5.2

PM92447 From the APAR:

The property "sib.msgstore.jdbcFailoverOnDBConnectionLoss" is enabled in v8.5 by default which implies that the server must be terminated when the connection to the database is lost, but it is not honored even when "sib.meEnableInstanceOnFailure" is disabled.

Prior to this corrective action the termination of the server may not haveoccurred when connection to the message store database was lost. With the corrective action the property is checked and properly enforced.

Two notes on this: (1) This corrective action is a necessary prerequisiteto the proper synchronization of components during server startup, which is the central point of PM76557 and the new settings it provided; and (2) The termination of the server when the ME database connection is lost is particularly useful in a clustered environment wherethe ME configuration is in an “active/passive” mode with automatic failover. If the message store connection is lost in server A, the ME in that server has compromised functionality. The High Availability Manager (HAM) will failover the ME function to server B, and terminating server A insures no additional work is taken on by that server. It's a clean failover scenario.

7.0.0.338.0.0.98.5.5.2

Additional control functions with PM76557

The following custom properties and variables were introduced with APAR PM76557. Each is described on the page indicated:

Custom Property or Variable Values Page

synchWithControlRegion true or false 9

sib.property.isMECritical true or false 9

adjunct_region_start_synchronized 0 or 1 10

adjunct_region_start_sib_waittime Time in seconds 10

adjunct_region_start_sib_abend 0 or 1 11

Custom property to synchronize a scheduler start with the overall server start

As discussed earlier, the focus here was the possibility of a configured scheduler to start and begin initiating work for the server before the server itself was fully started. Depending on the scheduler and the nature of the work it does, this may or may not be an issue. To correct scenarios where it was an issue an additional custom property was created:

15 The APAR PM76557 text does not mention the Scheduler issue, but the corrective action for that issue is included.




Property: synchWithControlRegion = true | false

Description: A value of true indicates the scheduler should not initiate any work until the server in which it operates is marked as “open for e-business.” In other words, the scheduler will wait until all the other functional components of the server are initialized and the server as a whole is fully started.

Default: If this property is not set, the default is false.

Location: Resources Schedulers Scheduler Custom properties

Then a new custom property is created with the appropriate setting. For example:

Schedulers are resources scoped to whatever level you wish (cell, node, server or cluster). Any given scheduler ends up being instantiated in an application server based on the deployed applications and their making use of the scheduler. Therefore, setting this new custom property for a scheduler scoped at cluster will apply to the scheduler instances in every member of the cluster.

Custom property and WebSphere variables related to ME start and server start

The focus here was the possibility of a server being “open for e-business” and accepting work before Messaging Engines hosted in the server's CRA were fully initialized. Depending on the nature of the work being done by applications in the server, that may or may not be an issue. To correct scenarios where it was an issue an additional custom property was created as well as three new variables.

Property: sib.property.isMECritical = true | false16

Description: This custom property provides a way to indicate which Messaging Engines17 must be active before the server in which it runs may considered started.

Note: this custom property by itself does not control the behavior. The behavior is controlled with the adjunct_region_start_synchronized variable, which is discussed next. This custom property simply provides a mark which MEs should be considered critical18 and thus subject to the control of the variable. Coding just this custom property and nothing else results in the same behavior as if this custom property wasn't coded at all.

Default: If this property is not set, the default is false, which means the ME is not marked as critical.

16 The default is false so leaving this property unset achieves the same as coding the property with false. However, if you code the property with false you will also receive a CWSID0082I information message.

17 A WAS environment may have several Messaging Engines. And in fact any given server may have more than one (though that's a little less common). This custom property is a way to indicate which MEs are to be marked critical.

18 Again, a critical ME is a Messaging Engine that must be fully active before the server is allowed to be marked “open for e-business.”




Location: [server or cluster] Messaging engines SIBus name Custom properties19

Then a new custom property is created with the appropriate setting. For example:

The APAR also provides three WebSphere variables that provide control over operational behavior for those servers in which there is an ME marked as critical.

The first variable determines whether the server “open for e-business” is issued only after the critical MEs are started:

Variable: adjunct_region_start_synchronized = 0 | 1

Description: For servers with an ME marked as critical, a value of 1 indicates the server's overall “open for e-business” status is held until the critical MEs come fully active.

If this variable is not set, or set to 0, then the overall server “open for e-business” may occur before the MEs are fully active20.

Default: If this property is not set, the default is 0.

Scope: The scope of this variable may be set to whatever level you wish to include the serverswith critical MEs. For example, a scope of cell implies all servers; a scope of cluster implies only those servers in the cluster.

The next variable determines how long a server should wait for critical MEs to come active before deciding it has waited long enough, and for the server to take other action:

Variable: adjunct_region_start_sib_waittime = time in seconds

Description: For servers with an ME marked as critical and the start_synchronized variable set to a value of 1, this value determines how long the server waits for the critical MEs to comeactive before taking other action. The other action is defined by the sib_abend variable, which is described next.

Default: If this property is not set, the default is 300 seconds.

Scope: Set the scope compatible with the scope implied by the servers with MEs marked as critical. Setting this scope equal to the scope provided for start_synchronized variable aligns the scope for related variables.

Note: A value of 0 indicates a special case where no timer is in effect.

19 There are two ways navigate to the custom properties for a messaging engine: as shown in the diagram above (through the properties of the server, or if clustered then the cluster definition), or: Service Integration x Buses Bus Name Messaging Engines Bus Name Custom Properties.

20 We emphasize the word may here because it's possible the timing will work out as desired and the MEs come active before the server is marked “open for e-business.” Without this new control function it's possible the MEs may or may not yet be quite ready. With this new control function the desired behavior – server “open for e-business” only after critical MEs active – is assured.




The final variable determines what action should be taken when the timer value set by the sib_waittime variable has elapsed:

Variable: adjunct_region_start_sib_abend = 0 | 1

Description: Defines what action should take place when a server with an ME marked critical and the start_synchronized variable set to 1 and the sib_waittime value (specified or implied) has expired:

0 – Issue a write to operator (WTO) message but take no other action1 – Issue WTO message and abend the entire server (CR, CRA and SRs)

If 1 is specified and the server is abended, then restart of that server may be performed by your operations staff or by automation routines.

Default: If this property is not set, the default is 0, which means issue WTO but do not abend the CRA.

Scope: Set the scope compatible with the scope implied by the servers with MEs marked as critical. Setting this scope equal to the scope provided for start_synchronized variable aligns the scope for related variables.

General Recommendations for new settings

The following are general recommendations. Your specific environment may indicate a differentapproach. As a reminder, here are the five new settings and what they relate to:

Custom Property or Variable Values Related to

synchWithControlRegion true or false Startup of scheduler function in server

sib.property.isMECritical true or false Startup of ME function in CRA

adjunct_region_start_synchronized 0 or 1 Startup of ME function in CRA

adjunct_region_start_sib_waittime Time in seconds Startup of ME function in CRA

adjunct_region_start_sib_abend 0 or 1 Startup of ME function in CRA

• If you are not experiencing any issues related to startup of function within your server environment, then you may not need to concern yourself with these new settings.

• The scheduler setting of syncWithControlRegion=true is somewhat unrelated to the other settings for the ME. A scheduler with this custom property will delay issuing work until the CR is “open for e-business.” However, the CR “open for e-business” may be held up waiting for a critical ME to initialize, and thus the scheduler will be held up as well. Functionally the scheduler settings and the ME settings are separate, but from a behavior perspective the isMECritical=true setting coupled with the start_synchronized=1 setting may influence the scheduler setting outcome.

• If your WAS z/OS environment has no CRA address spaces, then the last four properties in the table above do not apply to you. No CRA implies no Messaging Engines.

• If you code the sib.property.isMECritical = true for a Messaging Engine, then also code adjunct_region_start_synchronized=1 if you wish to delay the server start until the MEs are initialized. The isMECritical property by itself results in no action taken.

• As a “for your information” (FYI): if the ME start_synchronized=1 is in effect then the overall server start is delayed until all the critical MEs are initialized. That means any non-JMS work (such as HTTP traffic) will not be processed until the critical MEs initialize and the CR listener ports open.




• The default sib_waittime value of 300 seconds may be acceptable for your environment. But several things can influence whether 300 seconds is sufficient:

1. The timer is waiting for the configured minimum number of servant regions to come active. If you have a high number of configured minimum servant regions then it is possible the timer will expirebefore all the servants are active21.

2. If you have a high number of destinations configured in the Messaging Engine, then initialization of that ME may take longer than the default 300 seconds.

This value should be set to a value high enough to allow all the servant regions to initialize. The default of 300 seconds may accommodate this and work well. Be careful setting it lower, particularly lower than the time it takes for the configured minimum servants to initialize.

• The setting of adjunct_region_start_sib_abend=1 will result in the entire server (CR,all SRs and the CRA) being abended when the sib_waittime timer expires. The server will not be automatically restarted unless your system automation routines detect and restartthe server.

Conversely, a setting of adjunct_region_start_sib_abend=0 may result in what appears to be a hung server if an ME marked “critical” fails to start22. The server address spaces may be present but the CR listener ports will remain closed. The server is not “openfor e-business”.

• Coding a sib_waittime value of 0 results in no timer in effect. If sib_abend=1 and a sib_waittime=0 set means you will never see server abended if an ME fails to start. Be careful with sib_waittime value of 0.

21 This is particularly true on test systems where available resources may be limited.22 Or, fails to start in the time you expect it to start. The ME may in fact still be coming active but taking longer than expected.




Document Change HistoryCheck the date in the footer of the document for the version of the document.

May 22, 2014 Original document

May 23, 2014 Techdoc number updated on PDF and republished.

End of WP102450



default messaging apars - ibm application server for z/os ... and forwarding function very similar...

Documents