hnd202 - ppt

8/12/2019 HND202 - PPT

1/41

HND202 Using NSD: A Practical GuideRob Gearhart Domino Quality & Serviceability Engineer

Elliott Harden Field Support Engineer

8/12/2019 HND202 - PPT

2/41

Agenda

What is NSD?

NSD Major Sections

Call Stacks Memcheck

NSD Checklist

Case Studies

8/12/2019 HND202 - PPT

3/41

Agenda

What is NSD?

NSD Major Sections


NSD Checklist

Case Studies

8/12/2019 HND202 - PPT

4/41

What is NSD?

NSD (Notes System Diagnostic) is one of the primary FFDC diagnosticsused for Lotus Domino Products:

Domino/Notes

Quickplace/ DomDoc/ Domino Workflow

Sametime

FFDC = First Failure Data Capture

Used for troubleshooting Notes/Domino and Companion Products

Crashes Hangs

Severe Performance Problems (not a good tool for mild or moderate performance)

NSD applies equally well for Notes Client vs. Domino Server

8/12/2019 HND202 - PPT

5/41

What is NSD?

Used on all Platform's (except Mac)

ND6 - iSeries NSD is a different animal (output wise)

ND7 - iSeries NSD matches other platforms more closely

In 6.5.4, NSD on iSeries includes Memcheck (must be configured viaenvironment variable)

On Unix NSD is a shell script (nsd.sh)

Memcheck is a separate compiled binary

On W32 NSD is a compiled binary (nsd.exe)

Memcheck is built into nsd.exe

8/12/2019 HND202 - PPT

6/41

NSD Update Strategy

NSD is one of the primary focuses for Serviceability hasundergone many continual improvements

IBM has implemented the NSD Update Strategy to periodically

compile the newest improvements for NSD for the most recentexisting versions of Domino (ND6 & ND7)

Supported by the addition of versioning information for NSD

Special Hotfix installer for NSD (does not conflict with regular hotfix installer)

Periodically re-sync NSD source from ND8 back into MRs for ND6 & ND7

Current NSD Build is 2382 equivalent to ND7.0.2 version backported for numerous versions of Notes/Domino.

8/12/2019 HND202 - PPT

7/41

NSD Update Strategy

NSD Updates available to customers through 2 methods:

Contact Support or PSM to get hotfix installer

OR

Download from the website

For more information, see

Technote # 1233676 NSD Fix List and NSD Update Strategy(Fixlist here)

http://www.ibm.com/support/docview.wss?uid=swg21233676

Technote # 4013182 Updated NSD For Domino Releases (Downloads here)http://www.ibm.com/support/docview.wss?uid=swg24013182

8/12/2019 HND202 - PPT

8/41

Agenda

What is NSD?

NSD Major Sections


NSD Checklist

Case Studies

8/12/2019 HND202 - PPT

9/41

NSD Major Sections

Process Information (Call Stacks)

tells Support code path involved in the problem

Memcheck (Domino Memory Objects) tells Support resources (databases, files, views, users) involved in the problem

System Information

tells Support OS configuration (Patches, etc) NOT IN THIS LAB

Environment Info

tells Support execution environment (Notes INI, etc) NOT IN THIS LAB

8/12/2019 HND202 - PPT

10/41

NSD - Call Stacks

Dump of thread stacks for all Domino processes

including external applications that Call into Notes API

Provides insight to the code path where a crash or hang occurs

8/12/2019 HND202 - PPT

11/41

NSD - Call Stacks

W32 - for fatal thread, will make 3 passes

1). Dumps complete call stack (divided into "before" and "after" frames)

2). Granular break down of stack frames, showing arguments, returnaddress, basic register information

3). Function parameters that are pointers are de-referenced

UNIX Provides one pass for call stack

no break down of stack frames

register information for limited platforms (AIX, Linux & OS390)

On AIX - arguments may show as "???", meaning code not compiled withdebug levels

8/12/2019 HND202 - PPT

12/41

W32 Call Stacks

NSD takes three passes of the fatal call stack

Pass ONE dumps stack trace summary, but no frame info

Pass TWO dumps contents of stack frames (along with ascii equivalent)

Pass THREE de-references pointer parameters, meaning we can see thecontents of pointer arguments passed to a function

8/12/2019 HND202 - PPT

13/41

W32 Call Stacks - Pass ONE

Pass ONE - there are two halves of the call stack

First half of call stack is everything that happened AFTER the fatal (i.e. this iswhat the thread did to handle the exception).

Often times will see JVM_FindSignal near the lower portion. Ignore this, it isnothing.

############################################################

### t hr ead 5/ 21: [ nI MAP: 07b4: 06cc]

### FP=07e4e208, PC=77f 83786, SP=07e4e1e4, st kbase=07d50000, st ksi ze=262144

############################################################

[ 1] 0x77f 83786 nt dl l . ZwWai t ForSi ngl eObj ect+11 ( 560, 36ee80, 0, 601a7c06)

[ 2] 0x77e87837 KERNEL32. Wai t ForSi ngl eObj ect +15 ( 7e4e5a0, 77e8ae88, 7e4ec0c, 0)

@[ 3] 0x601a7046 nnot es. _OSFaul t Cl eanup@12+342 (0, 0, 0, 7e4ec0c)

@[ 4] 0x601b07b1 nnot es. _OSNTUnhandl edExcept i onFi l t er@4+145 ( 7e4ec0c, 7e4ec0c, 6ef 1ab5, 7e4ec0c)

[ 5] 0x1000e596 jvm._JVM_FindSignal@4+180 (7e4ec0c,77ea18a5,7e4ec14,0)

[ 6] 0x77ea8e90 KERNEL32. Cl oseProf i l eUser Mappi ng+161 ( 0, 0, 0, 0)

8/12/2019 HND202 - PPT

14/41

W32 Call Stacks - Pass ONE (cont)

Second half is what you are interested in (real meat of crash). Look formodule names, and function names

Look for FATAL thread

Fatal, panic, halt, access violation

Should pair this with console output (e.g. PANIC message)

W32 - NSD demangles C++ functions in the call stack, meaning itprovides the class name and function name (in that order)

8/12/2019 HND202 - PPT

15/41

UNIX Call Stacks

On Unix, there is only one pass (no dump of stack frame contents)

Upper portion of the call stack is the part of the stack that deals with thefatal condition

Look at portion of stack below the fatal, raise.raise, signal handler,abort, or terminate line

Which one this shows under depends on platform and nature of fatal

On Unix, C++ function names are mangled (except zSeries)

8/12/2019 HND202 - PPT

16/41

Agenda

What is NSD?

NSD Major Sections


NSD Checklist

Case Studies

8/12/2019 HND202 - PPT

17/41

NSD - Memcheck

Analyzes Domino Objects

Steps through shared and private pools allocated by Domino MemoryManager

Summarizes Memory Usage

Dumps information about Open Databases, Views and Documents, andOpen Files (in a nutshell)

Memory Usage does NOT include externally allocated memory, such asLotusScript, Java, or third-party code

Will need OS diagnostics to determine the total memory usage

8/12/2019 HND202 - PPT

18/41

NSD - Memcheck

Memcheck can be thought of as 3 major sections

Shared Memory

Private Memory

Resource Usage Summary

8/12/2019 HND202 - PPT

19/41

Shared Memory Includes

Summary of Shared Pools

KEYWORD "Shared Memory" - Total Shared Memory Usage should be around 1.1 GB

KEYWORD "Top 10" - largest block type should be UBM (0x82cd) at 750 MB

OS Package Info ND6 KEYWORD "Shared OS Field

ND7 KEYWORD MM/OS Structure Information

Indicates thread ID of crashing thread and PANIC Message (if any)

NSF Package Info KEYWORD "Open Databases" (lists db name, db handle)

KEYWORD "Open Documents" (lists noteID's and database handles)

NIF Package Info

KEYWORD NIF Collections" (lists open views)

KEYWORD NIF Collection USers" (lists [thread] users of those views)

8/12/2019 HND202 - PPT

20/41

Shared Memory Pool Summary

Shared Memory Stats ( Ti me 17: 45: 55) - - - - @@>

TYPE : Count SIZE ALLOC FREE FRAG OVERHEAD %used %f r ee

Static-DPOOL: 35 125829120 116479928 9334512 0 19408 92% 7%

Overall : 35 125829120 116479928 9334512 0 19408 92% 7%

Note Size shows overall amount of memory allocated by Domino MM, Allocshows whats actually in use (or sub allocated). You WANT %used to be high.

8/12/2019 HND202 - PPT

21/41

Top 10 Shared

Top 10 Shar ed Memor y Bl ock Usage . . . - - - - - - @@>

BY SI ZE

Type Tot al Si ze Handl es Typename

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

0x82cd 637673472 162 BLK_UBMBUFFER

0x8252 20971520 20 BLK_NSF_POOL

0x834a 18350080 18 BLK_GB_CACHE

0x82cc 10511340 161 BLK_UBMBCB

0x824b 9810466 160 BLK_OPENED_NOTE

0x8a03 6760070 1604 BLK_NETBUFFER

0x8311 5242880 5 BLK_NI F_POOL

0x890b 4578420 70 BLK_EXECPOOL

0x8a05 2460000 1 BLK_NET_SESSI ON_TABLE

0x8a01 2289210 35 BLK_NETPOOL

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

8/12/2019 HND202 - PPT

22/41

MM/OS Section

MM/OS Structure Information ( Ti me 13: 15: 45) - - - - - - @@>

Star t Ti me = 12/ 13/ 2005 01: 15: 02 PM

Cr ash Ti me = 12/ 13/ 2005 01: 15: 32 PM

Error Message = PANIC: LookupHandle: handle out of range

SharedDPool Si ze = 4194304

Faul t Recover y = 0x00010013

Cl eanup Scr i pt Ti meout= 300

Cr ash Li mi t s = 3 cr ashes i n 5 mi nut es

StaticHang = [ nhttp: 2752: 10]/[ nhttp: 2752: 3500] (0xac0/0xa/0xdac)

Conf i gFi l eSem = ( SEM: #0: 0x010d) n=0, wcnt=- 1, User s=- 1, Owner =[ 0: 0]

FDSem = ( RWSEM: #11: 0x410f ) r dcnt =- 1, r ef cnt =0 Wr i t er =[ 0: 0] , n=11, wcnt =- 1

8/12/2019 HND202 - PPT

23/41

Open Databases

Open Dat abases ( Ti me 11: 45: 21) - - - - - - @@>

D:\Lotus\DominoR65\Data\events4.nsf

Versi on = 43. 0

Si zeLi mi t = 0, War ni ngThr eshol d = 0

Repl i caI D = 86256ae0: 02697903

bCont Queue = NSFPool [ 0: 48836]

FDGHandl e = 0xf 01c0098, Ref Cnt = 10, Di r t y = N

DB Sem = ( FRWSEM: 0x0244) st at e=0, nl r dr s=0 Wr i t er =[ ]

SemContQueue = ( RWSEM: #0: 0x029d) r dcnt =- 1, Wr i t er=[ ] Owner=[ ]

By: [ nevent:0e2c: 2] DBH= 3, User=CN=Sithlord/O=SET




Note: edited for clarity some info is missing

8/12/2019 HND202 - PPT

24/41

Open Documents

Open Documents (BLK_OPENED_NOTE): total =352 . . . - - - - -- @@>

DBH NOTEID HANDLE CLASS FLAGS IsProf #Pools #Items Size Database

531 7330 0x24f f 0x0001 0x0200 Yes 1 4 2984 d:\notedata\drmail\jsmith.nsf

.

Open By: CN=J ohn Smi t h/ O=ACME/ C=US

Fl ags2 = 0x0404

Fl ags3 = 0x0000

Or i gHDB = 531

Fi r st I t em = [ 9471: 836]

Last I t em = [ 9471: 1228]

Non- pool si ze : 0

Member Pool handl e=0x24f f , si ze=2984

.

8/12/2019 HND202 - PPT

25/41

Note Classes

Replication Formula Note0x0800

Agent Note0x0200

ACL Note0x0040

View Note0x0008

Form Note0x0004

Data Note - document0x0001

Note TypeNote Class Value

8/12/2019 HND202 - PPT

26/41

Open Views NIF Collections

NIF Collections ( Ti me 12: 48: 35) - - - - - - @@>

CollectionVB ViewNoteID UNI D OBJ I D Ref Cnt Fl ags Opt i ons Corr upt Del eted Temp NS Ent r i esViewTitle

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

[ 0020e005] 1518 1356a8 358710 1 0x0000 00000008 NO NO NO NO 0MyNotices

CI DB = [ 0253cc05]

Col l Sem ( FRWSEM: 0x030b) st ate=0, wai t ers=0, r ef cnt=0, nl r dr s=0 Wr i t er=[ : 0000]

NumCollations = 2

bCol l ati onBl ocks = [ 001e72e5]

bCol l at i on[ 0] = [ 00117005]

bCol l at i on[ 1] = [ 001a2205]

Col l I ndex = [ 00012a09]

Col l at i on 0: Buf f er Si ze 26, I t ems 1, Fl ags 0

0: Ascending, by KEY, "StartDateTime", summary# 2

Col l I ndex = [ 00012c09]

Col l at i on 1: Buf f er Si ze 26, I t ems 1, Fl ags 0

0: Descending, by KEY, "StartDateTime", summary# 2

ResponseI ndex [ 0010e4b6]

Not eI DI ndex [ 0010e385]

UNI DI ndex [ 0010e5e7]

8/12/2019 HND202 - PPT

27/41

Open Views NIF Collection Users

NIF Collection Users (hash) (Time 12:48:33) ------ @@>

CollUserVB ... CollectionVB Remote OFlagsViewNoteID Data HDB/Full View HDB/Full ... Open By

------------ ... ------------ ------ ------ ---------- ------------- ------------- ... --------------

[ 00239805] ... [ 0023d005] NO 0x0082 786 1219/1874 1219/1874 ... [ nserver:09d8:04ca]

CurrentCollation = 0

[ 0013a805] ... [ 00136005] NO 0x00c2 11122 886/785 886/785 ... [ nserver:09d8:0266]


[ 0028d805] ... [ 0020e005] NO 0x00c2 1518 551/1432 551/1432 ... [ nserver:09d8:03b0]


8/12/2019 HND202 - PPT

28/41

Private Memory Includes

Info for each process

KEYWORD "Attach to process [procname:PID]" - to find beginning of info for each process

TLS Mapping

KEYWORD "TLS Mapping" - shows map of physical thread to virtual thread (its a Support"Thing")

Open Documents

KEYWORD "Open Documents" - lists documents opened in private memory (if any)

Private Pools Allocated through Domino Memory Manager KEYWORD "Process Heap Memory" - total size across all private pools (should be below 100

MB with a few exceptions like server & http)

KEYWORD "Top 10" - shows highest block type used

8/12/2019 HND202 - PPT

29/41

TLS Mapping

- - - - - - TLS Mappi ng - - - - -

NativeTID VirtualTID Pr i mal TI D

[ nSERVER: 0514: 0510] [ nSERVER: 0514: 0002] [ nSERVER: 0514: 0002]


[ nSERVER: 0514: 05d4] [ nSERVER: 0514: 0005] [ nSERVER: 0514: 0005]




Memcheck - prints out virtual thread ID in most places. We need to beable to map this to physical thread ID from the call stack. TLS Mappingsection does this quite nicely!

8/12/2019 HND202 - PPT

30/41

Open Documents (Private)

Open Documents (BLK_OPENED_NOTE): t ot al =352 . . . - - - - - - @@>

DBH NOTEID HANDLE CLASS FLAGS IsProf #Pools #Items Size Database

531 7330 0x24f f 0x0001 0x0200 Yes 1 4 2984 d:\notedata\drmail\jsmith.nsf

.

Open By: CN=J ohn Smi t h/ O=ACME/ C=US

Fl ags2 = 0x0404

Fl ags3 = 0x0000

Or i gHDB = 531

Fi r st I t em = [ 9471: 836]

Last I t em = [ 9471: 1228]

Non- pool si ze : 0

Member Pool handl e=0x24f f , si ze=2984

.

8/12/2019 HND202 - PPT

31/41

Top 10 Process Memory

Top 10 [ nSERVER: 09d8] Memory Bl ock Usage. . . - - - - - - @@>

BY SI ZE

Type Tot al Si ze Handl es Typename

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

0x4129 20447232 39 BLK_LOCAL

0x0a04 10595772 162 BLK_NET

0x028b 3327954 53 BLK_FOLDERREPLOPS

0x0910 1999180 1126 BLK_SRV_NAMES_LI ST

0x093c 1219526 242 BLK_SRV_HASH_TBL

0x024b 1131334 19 BLK_OPENED_NOTE

0x0221 930818 96 BLK_NEW_NOTE

0x0130 562418 1545 BLK_TLA

0x0149 548834 101 BLK_PHTCHUNK

0x030a 319190 1 BLK_LOOKUP_THREAD

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

8/12/2019 HND202 - PPT

32/41

Process Heap Memory

Process Heap Memory Stats ( Ti me 17: 46: 00) - - - - - - @@>

TYPE : Count SIZE ALLOC FREE FRAG OVERHEAD %used %f r ee

Static-DPOOL: 12 6291456 3795788 2489080 0 9486 60% 39%

VPOOL : 2 130808 8994 117628 0 4210 6% 89%

POOL : 3 86348 58790 24468 0 3114 68% 28%

Over al l : 12 6291456 3653692 2631176 0 16810 58% 41%

8/12/2019 HND202 - PPT

33/41


Provides great value (one stop-shopping)

KEYWORD "Resource Usage" - easy to read summary of open resourceslisted by process and thread (physical/virtual)

Lists Resources in use by each thread Open Databases (name and handle)

Open Views (name and handle)

Open Documents (noteID)

Open Files (OS file descriptor)

Search on the Physical Thread ID in question

KEYWORD "VThread [ ] Mapped To: PTHREAD [ ]"

8/12/2019 HND202 - PPT

34/41

Resources Per Thread

** VThread [ ndiiop:0904: 14]

. Mapped To: PThread [ ndiiop:0904: 1508]

. . usi ng: Pr i mal Thr ead [ ndi i op: 0904: 7]

. . SOBJ : addr =0x4cd7261c, h=0xf 010404d t =f 982 ( PKG_NSF9+386)

. . SOBJ : addr =0x51cd039c, h=0xf 0104043 t =c275 ( BLK_NSFT)

. . SOBJ : addr=0x4d4aba64, h=0xf 010404c t =c130 ( BLK_TLA)

. . Database: e:\notes\data\a_dir\archive.nsf

. . . . DBH: 3401, By: CN=J ohn Smi t h/ OU=New York/ O=ACME

. . . . DBH: 3664, By: CN=John Smith/OU=New York/O=ACME

...... view: hCol=3666, cg=N, noteID=798, (archiveLookup)|archiveLookup

. . . . DBH: 3665, By: CN=J ohn Smi t h/ OU=New York/ O=ACME

. . file: fd: 2388, e:\notes\data\a_dir\software_functions.nsf

8/12/2019 HND202 - PPT

35/41


Allows Support to quickly isolate any potential patterns regardingdatabase and/or documents

CAVEATS

Problem may not be directly attributable to a specific database/view/document

Resource Usage is not guaranteed to be a silver bullet

Just because a crash occurs on a database or document does NOT mean its thedatabase/document's fault (dont assume database corruption)

Never look at an NSD in a vacuum - must know the nature of the problem first, then use NSD to

fill in the gaps Must use insight from the call stacks and other key factors to know if this will determine a pattern

8/12/2019 HND202 - PPT

36/41

Agenda

What is NSD?

NSD Major Sections

Call Stacks

Memcheck

NSD Checklist

Case Studies

8/12/2019 HND202 - PPT

37/41

NSD Checklist - Call Stacks

Find the call stack for the crashing process/thread KEYWORDs FATAL, CHILD_DIED, HALT, PANIC

What is the physical thread ID? (you will need this later)

What was the crash point? (you will need Support's assistance)

What modules are on the stack? (NNOTES, NLSXBE, etc)

Is third-party code involved? (LSX, DSAPI, RDBMS Kernel, etc)

Symptoms What is the flow of events?

When did the crashes start?

What changed since it began?

How does the crash manifest itself (access violation, PANIC, etc) What do users experience?

How many servers/users affect

What do OS diagnostics show (CPU, disk, memory, etc)

You know the drill

8/12/2019 HND202 - PPT

38/41

NSD Checklist - Memcheck

Shared Memory Total Shared Memory Usage

Top 10 Shared Block Usage

Open Databases

Open Documents

Private (look at appropriate Process) Process Heap Memory

Top 10 Process Block Usage

TLS Mapping (if needed)

Resource Usage (find appropriate Physical Thread) Databases/Views/Documents

8/12/2019 HND202 - PPT

39/41

Agenda

What is NSD?

NSD Major Sections

Call Stacks

Memcheck

NSD Hit List

Case Studies

8/12/2019 HND202 - PPT

40/41

Agenda

Scenario 1 Agent Manager Crash (find agent note)

Scenario 2 Domino Server Crash (find out why)

Scenario 3 HTTP Crash (examine call stacks)

Scenario 4 HTTP Performance Problem (look at SEMDEBUG)

Scenario 5 (BONUS) Domino Server crash (find out why)

8/12/2019 HND202 - PPT

41/41

Discussion/Questions

[email protected]

[email protected]

NSD Knowledge Collection Technote #7007508 Knowledge Collection: NSD for ND 6 & 7


For NSD Update Strategy, see

Technote # 1233676 NSD Fix List and NSD Update Strategy(Fixlist here)http://www.ibm.com/support/docview.wss?uid=swg21233676

Technote # 4013182 Updated NSD For Domino Releases (Downloads here)
