agenda key: 55ch session number: 540038 performance analysis using idoctor job watcher dawn may...

8 Copyright IBM Corporation, 2009. All Rights Reserved.This publication may refer to products that are not currently available in your country. IBM makes no commitment to make available any products referred to herein.

Agenda Key: 55CHSession Number: 540038

Performance Analysis using iDoctor Job Watcher

Dawn [email protected]

2© 2009 IBM Corporation

IBM iDoctor for i

a rapid snapshot taker that allows real-time, detailed analysis

an IBM i exclusive!Patented technology!

Job Watcher


Job Watcher

the goals

Broaden the user base for Performance Investigationenable Operators, Programmers, IS Managementas well as Performance Specialists, Consultants

Simplify and automate processes

Provide quick, immediate access to collected data

Provide more analysis options


Job Watcher Performance Data Collector

What is Job Watcher

a snapshot approach a non-intrusive collector collects one or multiple jobs/tasks or all jobs/tasks harvests job details such as waits, call stacks by default optionally retrieve SQL, activation group, communication details analysis can start after just two snapshots

More…


Job Watcher Graphical User Interface

What is Job Watcher cont…

Windows Graphical Client application wizard for constructing the collection options viewer for managing the collections viewer for analyzing collections (during and post) currently a 106 predefined graphs setup a monitor to have JW running continuously

user defined query and user defined graph facilitiesuser can create queries and graphssave, categorize and transfer

data viewer can also be used against other DB files like collection services


Job Watcher

Starting Job Watcher

GUI startup wizard or green screen command select active jobs/threads/tasks from a GUI list

even jobs on a job queue set the collection options:

interval durationwatch durationwatch name and librarywatch descriptionwhat details to collectoptionally set a trigger to control when the watch should

start or what program to call if specified conditions are met during a watch (a trigger wizard is provided)


iDoctor vs. IBM i 6.1 Performance Tools

When to use iDoctor vs. the new tools?

General positioning:

• Basic, high-level performance management – use new tools

• Advanced, detailed analysis – continue to use iDoctor

October 2008 IBM Systems Magazine article comparing iDoctor vs. IBM i 6.1 Performance Tools http://www.ibmsystemsmag.com/ibmi/october08/trends/21990p1.aspx

Job Watcher


Job Watcher

waiting

Is there a problem?What potential for improvement is there?What areas would we have to address to improve performance?

What if a job's run/wait signature looked like......

running

running waiting

the first step in detailed Performance Investigation


Job Watcher

What if the job's run/wait signature looked like......

running waiting

DASDexample #1

Blocksexample #2

CPUqexample #3

- easy to use and non-intrusive to jobs being watched.- allows real-time, summarized or detailed views of how jobs spend time


Job Watcher

snapshot #1 snapshot #2

OK if idle wait (like Key/Think time)

But if a long DASD request or a block? (and the current wait has not ended yet)

current wait

current wait can span snapshot intervals


Job Watcher

snapshot #1 snapshot #2

the current wait “bucket values” should be investigated for “bad waits” also

for example, a large “bad wait” bucket time and count suggest many small duration waits are occurring

current wait

or they can be of very short duration

current wait bucket totals


T I M E

JW – use rule/action and the lurk facility with history

the problem occurs

satisfied Job Watcher "rule" starts up DB recording

start investigation

JW coverage

dump reduction start investigation


Job Watcher

Job Watchera real-time, high-performance, snapshot sampler

created to fill the gap between

Collection Services

- and -

the short duration, high data volumes, long dump/analysis times of PEX Trace

Job Watcher in a nutshell


Once installed, we are ready to begin.


Connect to your system.


Launching into the world of Job Watcher


Job Watcher Demos

1. Starting a Job Watcher collectionSetting up a Job Watcher monitor

2. Job Watcher analysisObject Lock Contention

3. Setup a collection using the trigger option4. Job Watcher analysis and Situational Analysis


Demo #1

Starting a Job Watcher CollectionSetting up a Job Watcher Monitor


Working with Job Watcher Monitors

Click on “Job Watcher” then select “Monitors”.

Monitors are in the tree-view similar to collections.


Refresh button


All existing collections will be displayed


And many hours later….


Demo #2

Job Watcher Data AnalysisObject Lock Contention


Demo #3

Job Watcher Data Analysis


Customer supplied examples of the problem

223:33:225/4/200523:33:205/4/2005327500FRGJOB

223:36:585/4/200523:36:565/4/2005327472FRGJOB

223:45:125/4/200523:45:105/4/2005327466FRGJOB

321:45:235/4/200521:45:205/4/2005327451FRGJOB

321:45:235/4/2020521:45:205/4/2005327447FRHJOB

321:54:065/4/200521:54:035/4/2005327455FRHJOB

321:54:065/4/200521:54:035/4/2005327461FRGJOB

419:59:295/4/200519:59:255/4/2005327410FRGJOB

522:44:215/4/200522:44:165/4/2005327422FRGJOB

523:06:135/4/200523:06:085/4/2005327432FRHJOB

623:06:135/4/200523:06:075/4/2005327450FRGJOB

1123:46:375/4/200523:46:265/4/2005327446FRGJOB

secondsEndedEndedStartedStartedNumName

ElapseTimeDateTimeDateJobJob

Nightly processing log – 05/04/2005


Fast path to JW supplied graphs

Collection management

options

Collection details, system info.

Graph scope Available graphs


Start with a search for the job referenced…


Start with a search for the job referenced…

Enter job name and time range


Select an interval of a candidate and go for a run/wait signature…


Waiter’s run/wait signature


Legend on/off button


Menu Buttons Interval Arrows


Drill down for call stack, holder, etc. or go for a collection overview


Holder’s run/wait signature


Who is using CPU?


Why is job# 328268 burning so much CPU?


List other jobs that are also engaged in QQQTSORT…


Conclusion:

• Using Job Watcher we were able to determine that there were a number of jobs running sorts consuming high CPU. This caused jobs to wait excessively in CPUQ resulting in seizes to be held longer causing our job to wait for a seize much longer than expected.

• DBMON and/or Visual Explain should be used to review the SQL statements and the access methods being used in an effort to understand the reasons for the sort.


In Summary

The Job Watcher Approach

allows real-time and post-watch inspection of run/wait components

provides results for day to day use by coders, testers, and operations as well as for presentations in the executive boardroom

requires less skill to get started in performance investigation and makes the skilled, experienced performance investigator more productive


Demo #4

Job Watcher Data Analysis


Job Watcher analysis demo

• The investigation of a very large, busy 595 during a slowdown

• Use some of the newer iDoctor features

• Starting with Situation Analysis!


What have we learned so far about the data…

• Long DASD response experienced by jobs (mostly related to writes).

• Long DASD write response is slowing DBCLOSE operations, which in turn causes many users to experience long seize wait contention.

• The DBCLOSE operation is forcing index and data space pages to DASD (is this the last updater forces all the file indexes and data space pages out scenario?).

• Other significant overview wait categories (Disk Space Contention, Abnormal Contention and Disk Op-Start Contention) suggest pressure on DASD control blocks, stress in the create/destroy/extends of DASD space and waiting on system lists/queues to get a turn – all events closely related to file opens/closes, users signing on/off, new job creation (JW misses job fragments and short-lived jobs – so SQE could be a factor).

• There are lots of new jobs (inspection of initiations/termination – and the increase in TDE count from beg to end of collection. (see extra credit query example).

• Bursts of IO requests that JW can’t see (but would be caught in CSI reports along with DASD unit statistics to allow comparison of JW system oriented DASD response times to CSI hardware DASD response times).


And about Job Watcher features…

• Situation Analysis

• Alternate View support, Two graph support, Variable width and Normalize buttons, Legend options, Table support, Record Quick View

• Drill downs -- to call stacks, call stack program occurrence reports, object occurrence reports and Current wait details

• The iDoctor SQL editor

• Extra Credit -- a look at the JW Thread wait time signature that ran and collected the data for/during this collection

• Extra Credit -- a user defined query to count new the TDE’s created per interval


And about Current Waits…

• Job Watcher captures TDE accumulative data counters at interval snapshots … including the current wait condition.

JOB100010

JOB100009

JOB100008

JOB100007

JOB100006

Time - - - - - - - - - - - - - - - - - - - - ->

Interval Snapshot 19 20 21

Current Wait Reported at snapshot The actual Wait duration


Visit the web site for more information


®

262262 © 2008 IBM Corporation

Trademarks and Disclaimers8 IBM Corporation 1994-2009. All rights reserved.References in this document to IBM products or services do not imply that IBM intends to make them available in every country.Trademarks of International Business Machines Corporation in the United States, other countries, or both can be found on the World Wide Web at http://www.ibm.com/legal/copytrade.shtml.

Adobe, Acrobat, PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, other countries, or both.

Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce.

ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Cell Broadband Engine and Cell/B.E. are trademarks of Sony Computer Entertainment, Inc., in the United States, other countries, or both and are used under license therefrom.

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

Information is provided "AS IS" without warranty of any kind.

The customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here.

Prices are suggested U.S. list prices and are subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

agenda key: 55ch session number: 540038 performance analysis using idoctor job watcher dawn may...

Documents