implementing the output apar enhancements · wp102267 – implementing the output apar enhancements...

WebSphere on z/OS V7, V8.0, V8.5

Implementing the OutputImplementing the OutputImplementing the OutputImplementing the Output

APAR (APAR (APAR (APAR (PM74923)PM74923)PM74923)PM74923)

enhancementsenhancementsenhancementsenhancements

This document can be found on the web at:www.ibm.com/support/techdocs

Search for document number WP102267 under the category of "White Papers"

Version Date: April 11, 2013See "Document Change History" on page 3 for a description of the changes in this version of the document

Mike LoosConsulting IT [email protected]

(651) 270-3324

Written and provided by the WebSphere Application Server for z/OS team at the

Washington Systems CenterWashington Systems CenterWashington Systems CenterWashington Systems Center

IBM Advanced Technical SkillsIBM Advanced Technical SkillsIBM Advanced Technical SkillsIBM Advanced Technical SkillsGaithersburg, MD, USA

WP102267 – Implementing the Output APAR enhancements

The WebSphere Application Server for z/OS support team at the WashingtonSystems Center consists of: Don Bagwell, John Hutchinson, Mike Kearney, Mike Loos, Lee-Win Tai, and Louis Wilen.

Mike Cox, Distinguished Engineer, serves as technical consultant and advisor. Mike also was the person with the idea to produce this paper and was instrumental in the fine tuning of the information provided in it. He also contributed the graphic in the overview section.

Brian Pierce provides invaluable system programmingsupport to the team and considerable input to this paper in particular.

Dave Follis from z/OS WebSphere Development provided an interface with the developers during the development of the enhancement.

And a very special THANK YOU to Adam Wisniewski from z/OS WebSphere Development who actually made the changes to the product!

For questions or comments regarding this document, e-mail Mike Loos at

[email protected]

© 2013 IBM Corporation Americas Advanced Technical Skills - 2 -

ibm.com/support/techdocs Version Date: Thursday, April 11, 2013


Table of Contents

Overview.............................................................................................................4

Assumptions.......................................................................................................6

Creation of the Output Filesystem....................................................................7

Setting the Environment Variable.....................................................................9

Security System Setup.....................................................................................10

Create the HTTP Server....................................................................................11Initial HTTP server creation..........................................................................................................11

Modifications to the httpd.conf file..............................................................................................11

Create a JCL Procedure (PROC) to Run the Server....................................................................16

A Demonstration of Use...................................................................................17Switching the Active Output Files................................................................................................19

Document Change History...............................................................................21




OverviewThe handling of the output produced by the various address spaces associated with a WebSphere Application Server on z/OS (controller, servant(s), adjunct) has been addressed in the past by documents available on the techdocs website (http://www.ibm.com/support/techdocs). TD101087, and TD103695 both address the subject. The addition of the High Performance Extensible Logging facility in V8 added yet another option. The enhancement introduced with this APAR, is yet another, and provides additional capability that was not previously available.

It is likely that the function provided via this enhancement will be used in the development and test environment, where there are developers who may not have access to TSO and SDSF, or are otherwise not inclined to using TSO. The use of this enhancement is in no way precluded from production environments. Each installation will need to evaluate if, how, and where they wish to implement this enhancement.

The new function provided via this APAR (PM74923) is the addition of a single environment variable. The value associated with the variable is a path name which will be used to store the information which normally would be sent to the destination specified by the SYSPRINT DD card (STDOUT) and the SYSOUT DD card (STDERR). It also provides a way to seamlessly close the existing file and switch to a new file via the entry of a z/OS MODIFY command. The MODIFY command would allow the timely “rolling” of these files based on an installations needs, probably via regular automation.

The functionality provided in this APAR is better than simply coding PATH= on the SYSOUT and SYSPRINT DD statements to allow writing to the filesystem instead of the SPOOL.

1. The file is named with an established pattern.

2. There is capability provided to allow the spinning off of the file through the MODIFY command.

3. This captures ALL of the output, where the use of PATH= in the DD statements can allow the loss of some lines of output on occasion.

While the APAR addresses getting the files into the filesystem, this document describes an approach that enables the viewing of those files, associated with specific cells, nodes, clusters or servers to be viewed by designated groups of people in a simple manner, without providing access by those groups to the physical file system components.

This is done by exploiting the functionality of the APACHE HTTP server included in WebSphere on z/OS.




This document will show you how to:

1. Setup an Apache HTTP server which is a member of the SAF group(s) to which the cell components belong,

2. Map a hierarchy of URLs to a file system hierarchy,

3. Utilize SAF to control access to this hierarchy of URLs,

4. Provide read access to the authenticated and authorized users through the Apache configuration directives.

This is very easy to set up, and can often be accomplished in as little as an hour by following the steps outlined in this document (it actually takes longer to read the document than it does to set up the servers).

Upon completion, your developers/testers can now look at job logs without Telneting/SSHing into USS or using TSO and OMVS/ISHELL to view these server files or using TSO and a job log viewer like SDSF. All they need is a browser.

Use of this technique is easily extensible to other server specific files such as:

1. verbose garbage collection files if they are written to specific files rather than allowed to default to the SYSOUT DD statement.

2. FFDC directory contents as specified by the -Dcom.ibm.ffdc.log environment variable.

What this document will provide is step by step approach to implementing this new function.

In short, this will involve the following steps:

1. Creation of a filesystem to provide space for the output.

2. Setting the appropriate environment variable to enable the new function..




3. Some Security setup.

4. Simple instructions for creating a new HTTP Server V7 or later to allow access to the newly stored output.

5. Instructions for securing the output.

AssumptionsThe WebSphere Application Server must be at a maintenance level that implements the new function.

Version Maintenance Level

V7 7.0.0.29

V8R0 8.0.0.6

V8R5 8.5.0.2

The installation must have installed the FMID or Package that contains the HTTP Server at the appropriate level.

Version FMID or Package Name

HTTP Server V7 HHAP700

HTTP Server V8 com.ibm.websphere.IHS.zOS.v80

HTTP Server V8.5 com.ibm.websphere.IHS.zOS.v85

This document provides information on the setup of the Apache based HTTP server to access the information found by default in the //SYSPRINT and //SYSOUT DD files (STDOUT and STDERR). It does not provide information on how to setup the CERN based HTTP server which has shipped with z/OS for many years, often referred to as the Domino Go Webserver (DGW).

The implementation of this enhancement has the capability of generating many files of a significant size in the USS filesystem. The maintenance (archival, lifecycle, etc.) of those files is left as a user exercise. As mentioned, the capability of “rolling” the files is provided, but nothing more.




Creation of the Output Filesystem.If you have either WebSphere Application Server V8 or V8.5 installed, you may use the zCreateFileSystem.sh script provided with the Installation Manager used to install the products. An example batch job to run the script would look like this:

//CREATE EXEC PGM=IKJEFT01,REGION=0M //SYSTSPRT DD SYSOUT=* //BPXOUT DD SYSOUT=* //STDOUT DD SYSOUT=* //SYSTSIN DD * BPXBATCH SH + /Service/InstallationManager/bin/eclipse/tools/+ zCreateFileSystem.sh + -name WSCZFSLG.SHARED.WAS.OUTPUT.ZFS + -type ZFS + -volume ZFSMJ2 + -cylinders 6000 6000 + -mountpoint /shared/output + -owner MJADMIN + -group MJCFG /*

You'll note that the owner and group for the mountpoint are being set to the same as the owner and group of the application server's configuration filesystem. That will insure that the server will be able to successfully write to the filesystem.

On operating system releases prior to z/OS 1.13, the system ownership of the filesystem and the lpar upon which the application servers and the HTTP server executes should be the same for performance reasons.

On z/OS 1.13, there has been significant improvement in cross system write performance to the point where it makes no difference. So if you are running z/OS 1.13, then it is completely feasible to create a single, shared filesystem and mountpoint for multiple cells (nodes, servers) with no degradation in performance.

If you do not have access to the zCreateFileSystem.sh script, then the following commands issued from a USS shell will accomplish the same thing:

Assuming that /shared already exists;

mkdir -m 775 /shared/outputzfsadm define -aggregate WSCZFSLG.SHARED.WAS.OUTPUT .ZFS -cylinders 6000 6000 -volume ZFSMJ2mount -t zfs -f WSCZFSLG.SHARED.WAS.OUTPUT.ZFS /sha red/outputchmod 775 /shared/outputchown MJADMIN:MJCFG /shared/output




Next, you will need to create a “server specific” directory for each component for which you wish to collect output. The following commands illustrate that for all of the components in the “mjcell”.

mkdir -m 775 /shared/output/mjcell mkdir -m 775 /shared/output/mjcell/mjdemn mkdir -m 775 /shared/output/mjcell/mjdmnode mkdir -m 775 /shared/output/mjcell/mjdmnode/mjdmgr mkdir -m 775 /shared/output/mjcell/mjnodea mkdir -m 775 /shared/output/mjcell/mjnodea/mjagnta mkdir -m 775 /shared/output/mjcell/mjnodea/mjsr01a mkdir -m 775 /shared/output/mjcell/mjnodea/mjsr02a mkdir -m 775 /shared/output/mjcell/mjnodeb mkdir -m 775 /shared/output/mjcell/mjnodeb/mjagntb mkdir -m 775 /shared/output/mjcell/mjnodeb/mjsr01b mkdir -m 775 /shared/output/mjcell/mjnodeb/mjsr02b chown -R mjadmin:mjcfg /shared/output chmod -R 775 /shared/output

Now all that remains is to create a symbolic link from the cell to the shared output area.

ln -s /shared/output/mjcell /wasv7config/output/mjc ell

The use of a symbolic link allows us to redirect the output to another location without going through the process of modifying the WebSphere environment entries. For instance, if you wanted to temporarily go back to actual SYSOUT instead of the filesystem, you could just delete the link. If WebSphere cannot write to (or find) the directory specified for output it will revert to writing to the “default” location, SYSOUT.




Setting the Environment Variable.Setting the environment variable is what signals WebSphere Application Server to redirect the output from wherever the DD cards for SYSPRINT and SYSOUT point to the path specified by the environment variable. The variable name is: redirect_server_output_dir and the value assigned to it is a fully specified pathname to the location where the output will be written. The path must exist prior to use, and the Application Server must have sufficient authorization to write to it. To add the variable, login to the adminconsole. In the navigation panel on the left hand side, click on Environment, then on WebSphere variables. Set the scope for the component for which you wish to add the support. Click New to add a new variable.

Now you can add the variable. Set the Name to “redirect_server_output_dir”, and the Value to the path name, in our example, “/wasv7config/output/mjcell/mjnodea/mjsr01a”.

Then click OK and save and synchronize. The change will take effect at the next server restart. This step should of course be repeated for each component (cell, node, cluster, server) for which you wish to redirect the output.




Security System SetupIt is necessary to do a little security work prior to creating and starting the new HTTP server. We will be running the HTTP server under the same userid and group that is used for the file ownership. So we'll need a STARTED class profile. The name of the HTTP server will be MJHTTP01. The following command will create the necessary profile.

The HTTP server in this case can be thought of as just an additional server in the cell(s) and it must have access to read files created by the CR, SR, and Adjunct address spaces. That is the reasoning behind having it run under the admin id and group. If the same server is to be used to handle multiple cells, then it would be a simple matter to connect the admin id to the other cells' config groups.

RDEF STARTED MJLHTTP01.* STDATA(USER(MJADMIN) GROUP(MJCFG) TRACE(YES))

and we'll need to refresh the STARTED class in-storage profiles.

SETR RACLIST(STARTED) REFRESH

Also, the server will need access to the FACILITY class profile, BPX.SERVER. It may already exist, but if it doesn't it could be created with the following command.

RDEF FACILITY BPX.SERVER UACC(NONE)

Whether we have just created it or it existed previously, we can now give the group under which the HTTP server will run access to it.

PERMIT BPX.SERVER CLASS(FACILITY) ID(MJCFG) AC(READ )

And refresh the in-storage profiles.

SETR RACLIST(FACILITY) REFRESH

We can now create the GROUPs which will be used to provide access to the WebSphere Application Server's output directories. This will demonstrate a large degree of flexibility. You may need far less, or more likely just some portion of this, but we'll define a full range of groups to match the full range of output directories for demonstration purposes.

AG MJCELL OMVS(AUTOGID) /* Entire cell acc ess */AG MJDMNODE OMVS(AUTOGID) /* Entire dmgr nod e access */AG MJNODEA OMVS(AUTOGID) /* Entire mjnodea node access */AG MJNODEB OMVS(AUTOGID) /* Entire mjnodeb node access */AG MJSR01A OMVS(AUTOGID) /* Access to mjsr0 1a only */AG MJSR01B OMVS(AUTOGID) /* Access to mjsr0 1b only */AG MJSR02A OMVS(AUTOGID) /* Access to mjsr0 2a only */AG MJSR02B OMVS(AUTOGID) /* Access to mjsr0 2b only */

We'll do a little more security later, but the above will allow us to get started.




Create the HTTP Server.There are several ways to access the output that will be stored in the output location specified by the application server. You could browse it directly from a shell (telnet, ssh, OMVS, ISH, ISPF option 3.17), access it via an ftp client such as filezilla, use an SMB server to access it via Windows explorer, or via an HTTP server. The way that this document will demonstrate will be to set up a simple HTTP server and access the files directly. This allows us to use SAF (RACF) to secure the various output locations that might be available through simple membership in a RACF group.

Initial HTTP server creation.

The HTTP server that this document will demonstrate is the IBM HTTP Server V8.5 (apache) that is delivered with the WebSphere Application Server V8.5. A similar server with the same setup requirements is delivered with WebSphere Application Server V7.0 and V8.0.

The installation location can be anywhere in the filesystem to which there is appropriate access. This installation will be to /wasv7config/mjcell/mjhttp01 . The location of the HTTP server product is at /usr/lpp/IHSA/V8R5BASE .

To start:

cd /usr/lpp/IHSA/V8R5BASE/bin./install_ihs /wasv7config/mjcell/mjhttp01 80./shared/IHSA/V8R5BASE/bin/postinst -i /wasv7config /mjcell/mjhttp01 -t install -v PORT=80 -v SERVERNAME=wsya.wsclab.was hington.ibm.com chown -Rh mjadmin:mjcfg /wasv7config/mjcell/mjhttp0 1

This will create an HTTP server at the install location that will listen on ipname wsya.wsclab.washington.ibm.com port 80 (default) .

If the default port is already in use (highly likely), simply choose an unused port and replace 80 with it in the install_ihs command and the postinst command it generates.

The chown command sets the ownership of the server to be the same as the application server.

Modifications to the httpd.conf file.

It will also be necessary to make some modifications to the /wasv7config/mjcell/mjhttp01/conf/http.conf file to support SAF authentication.

The following LoadModule statements must be added or uncommented as appropriate.

LoadModule auth_basic_module modules/mod_auth_basic .so LoadModule authnz_saf_module modules/mod_authnz_saf .so LoadModule authz_default_module modules/mod_authz_d efault.so

If SSL support is required, then the following LoadModule statement must be uncommented.

LoadModule ibm_ssl_module modules/mod_ibm_ssl.so




The DocumentRoot directive should be changed to point at the base output location.

DocumentRoot "/wasv7config/output/mjcell"

The Directory statement should be changed from where it points by default, /wasv7config/mjcell/mjhttp01/htdocs to be the same as the DocumentRoot directive.

So change:

From:

# # This should be changed to whatever you set Docume ntRoot to.# <Directory "/wasv7config/mjcell/MJHTTP01/htdocs">

To:

# # This should be changed to whatever you set Docume ntRoot to.# <Directory "/wasv7config/output/mjcell"> SetOutputFilter DEFLATE CharsetSourceEnc IBM-1047 CharsetDefault ISO8859-1

Change the options directive:

From:

Options FollowSymLinks

To:

Options FollowSymLinks Indexes IndexOptions NameWidth=80

The following two changes are to support the suppression of an error message that Internet Explorer will throw when it sees a txt file with verbose garbage collection data. It believes it (correctly) to be an xml document, but since the verbose gc data may be incomplete, it sees it as malformed xml and won't display it. These two changes suppress that behavior.# # Uncomment the next line to allow MSIE verbose GC error suppression. # LoadModule headers_module modules/mod_headers.so

And:

# Added the following to suppress MSIE from throwin g error on verbose GC data.<Files *.txt> Header set X-Content-Type-Options nosniff




</Files>

Add the following Location directives.

The first one secures the HTTP server. It will cause any access to the server to cause a browser prompt for the user's ID and Password.

The second one secures the actual output location for an entire cell. It will only allow members of a group, in this case MJCELL (or of course members of the config group MJCFG) to see the output for location /mjcell. By including the MJCELL (and MJCFG) groups on subsequent location directives, we are effectively granting access to members of those groups to everything.

An additional location directive would be added for each additional output location for different nodes and servers within the cell, each with a group name that allows access to the output of that component, as well as the group name for the components above it in the cell hierarchy.

So, looking at the last location directive for output location /mjcell/mjnodeb/mjsr02b, the groups in the list are: MJCFG, MJCELL, MJNODEB, and MJSR02B. This allows access to those group members so that members of group MJSR02B have access to only this directory, while members of the higher level groups have access to either all server locations in the node, or to all locations in the entire cell.

The location directives are listed here:# # The following location directives control acces s: # # # This first one controls access to the server. # <Location /*> AuthName wasv7output AuthType Basic AuthBasicProvider saf Require valid-user AuthSAFExpiration "EXPIRED! oldpw/newpw/newpw" AuthSAFReEnter "Enter new password one more time" </Location> # # This following ones control access to various cel ls, # nodes, and servers. # <Location /mjcell> AuthName mjcelloutput AuthType Basic AuthBasicProvider saf Require saf-group MJCFG MJCELL AuthSAFExpiration "EXPIRED! oldpw/newpw/newpw" AuthSAFReEnter "Enter new password one more time" </Location>




<Location /mjcell/mjdmnode> AuthName mjcellmjdmnodeoutput AuthType Basic AuthBasicProvider saf Require saf-group MJCFG MJCELL MJDMNODE AuthSAFExpiration "EXPIRED! oldpw/newpw/newpw" AuthSAFReEnter "Enter new password one more time" </Location> <Location /mjcell/mjnodea> AuthName mjcellmjnodeaoutput AuthType Basic AuthBasicProvider saf Require saf-group MJCFG MJCELL MJNODEA AuthSAFExpiration "EXPIRED! oldpw/newpw/newpw" AuthSAFReEnter "Enter new password one more time" </Location> <Location /mjcell/mjnodea/mjsr01a> AuthName mjcellmjnodeamjsr01aoutput AuthType Basic AuthBasicProvider saf Require saf-group MJCFG MJCELL MJNODEA MJSR01A AuthSAFExpiration "EXPIRED! oldpw/newpw/newpw" AuthSAFReEnter "Enter new password one more time" </Location> <Location /mjcell/mjnodeb/mjsr01b> AuthName mjcellmjnodebmjsr01boutput AuthType Basic AuthBasicProvider saf Require saf-group MJCFG MJCELL MJNODEB MJSR01B AuthSAFExpiration "EXPIRED! oldpw/newpw/newpw" AuthSAFReEnter "Enter new password one more time" </Location> <Location /mjcell/mjnodeb> AuthName mjcellmjnodeboutput AuthType Basic AuthBasicProvider saf Require saf-group MJCFG MJCELL MJNODEB AuthSAFExpiration "EXPIRED! oldpw/newpw/newpw" AuthSAFReEnter "Enter new password one more time" </Location> <Location /mjcell/mjnodea/mjsr02a> AuthName mjcellmjnodeamjsr02aoutput AuthType Basic AuthBasicProvider saf




Require saf-group MJCFG MJCELL MJNODEA MJSR02A AuthSAFExpiration "EXPIRED! oldpw/newpw/newpw" AuthSAFReEnter "Enter new password one more time" </Location> <Location /mjcell/mjnodeb/mjsr02b> AuthName mjcellmjnodebmjsr02boutput AuthType Basic AuthBasicProvider saf Require saf-group MJCFG MJCELL MJNODEB MJSR02B AuthSAFExpiration "EXPIRED! oldpw/newpw/newpw" AuthSAFReEnter "Enter new password one more time" </Location>

Save the http.conf file.




Create a JCL Procedure (PROC) to Run the Server.

The server could be started from a shell, but in order to make it easy to set the userid and group under which the server runs, we'll run it as a started task using a PROC in PROCLIB.

Create a member in SYS1.PROCLIB (or whichever PDS is appropriate at your installation) called MJHTTP01. This member name must match the first part of the STARTED class PROFILE we created earlier (MJHTTP01.*).

The member should look like this:

//MJHTTP01 PROC ACTION='start', // DIR='/wasv7config/mjcell/mjhttp01/', // CONF='conf/httpd.conf' //*------------------------------------------------ --------- //IHS EXEC PGM=BPXBATCH, // PARM='SH &DIR/bin/apachectl -k &ACTION -f &CONF -DNO_DETACH',// MEMLIMIT=512M //STDOUT DD SYSOUT=* //STDERR DD SYSOUT=* //*STDOUT DD PATH='&DIR/logs/proc.output', //* PATHOPTS=(OWRONLY,OCREAT,OTRUNC), //* PATHMODE=(SIRUSR,SIWUSR,SIRGRP,SIWGRP) //*STDERR DD PATH='&DIR/logs/proc.errors', //* PATHOPTS=(OWRONLY,OCREAT,OTRUNC), //* PATHMODE=(SIRUSR,SIWUSR,SIRGRP,SIWGRP) // PEND




To start the server you would issue the following command on a console (or from whatever interface you may be using to emulate a console). Since the default action is 'start ', no parameters are necessary.

S MJHTTP01

To stop the server when it is running, simply issue the same command with an action='stop' parameter. The case (lower) in the action parameter is important.

S MJHTTP01,ACTION='stop'

A Demonstration of Use

Before starting, make sure that the server is running.

S MJHTTP01

Next, from a browser, access the server.

To start this we will authenticate with a userid which is NOT connected to the group which allows access to the output for mjcell.

As you can see on the next screen, we've attained access to the server, but not to any output locations.

Next we'll connect the userid, mjloos2 , to the group that allows access to the output location for the mjcell. This can be done with the following RACF command.

CONNECT MJLOOS2 GROUP(MJCELL)

After issuing the command, return to the browser screen and click on the “refresh” button.




We now can see the output location for mjcell. Clicking on it we now have a list of nodes.

Click on the mjnodea/ link to get a list of servers in the node.

Now click on the mjsr01a/ link to get a list of output files.

Now simply click on whichever output file you wish to see. We'll try,

MJCELL.MJNODEA.MJSR01A.MJSR01A.STC00782.CTL.121204. 153821.SYSPRINT.tx




And we see the output.

Switching the Active Output Files

It was mentioned earlier that it was possible to cause the application server to a new set of output files. The command to do that is an MVS Modify command (MODIFY or F). To switch to a new set of output files for server mjsr01a you would issue the following command at an z/OS console.

F MJSR01A,ROLL_LOGS

You can then click the refresh button from the server output selection page on your browser to see the new files.

© 2013 IBM Corporation Americas Advanced Technical Skills

- 19 - ibm.com/support/techdocs Version Date: Thursday, April 11, 2013


As you can see there are new logs with new date/time stamps for each component.

When WebSphere Application Server switches the logs, it writes an entry in the “old” log indicating which file name it is switching from, and to.

BBOO0408I OUTPUT REDIRECTED from: /wasv7config/output/mjcell/mjnodea/mjsr01a/MJCELL.M JNODEA.MJSR01A.MJSR01A.STC00782.CTL.121204.153821.S YSPRINT.txt BBOO0408I OUTPUT REDIRECTED to: /wasv7config/output/mjcell/mjnodea/mjsr01a/MJCELL.M JNODEA.MJSR01A.MJSR01A.STC00782.CTL.121205.111859.S YSPRINT.txt

It also writes an entry into the corresponding new file so that you will have a record of the chain of logs.

BBOO0408I OUTPUT REDIRECTED from: /wasv7config/output/mjcell/mjnodea/mjsr01a/MJCELL.M JNODEA.MJSR01A.MJSR01A.STC00782.CTL.121204.153821.S YSPRINT.txt

An added note of interest is that a similar message is written to the actual SYSOUT when the server is started to indicate that this function is in use, and to clarify why there is nothing else in the sysout.

This command could easily be used by automation software to roll the output files on a regular basis, such as everyday at midnight, or every four hours, or whatever is the correct interval for your installation.




Document Change HistoryCheck the date in the footer of the document for the version of the document.

April 8, 2013 Original document.

April 11, 2013 Updated with assigned WP102267 number, made several minor editing changes, and republished.

End of WP102267



implementing the output apar enhancements · wp102267 – implementing the output apar enhancements...

Documents