un [1999] solaris system performance management sa-400 student guide

8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide

1/698

Sun Microsystems500 Eldorado BoulevardMS: BRM01-209

Broomfield, Colorado 80021U.S.A.

SolarisSystemPerformance Management

Revision B, October 1999

SA-400

StudentGuide


2/698

PleaseRecycle

Cop yrigh t 1999 Sun Microsystem s, Inc., 901 San A nton io Road, Palo A lto, California 94303, U.S.A. All rights reserved .

This pro du ct or docu men t is pr otected by copy right and d istributed u nd er licenses restricting its use, copying ,

distribution, and decomp ilation. No p art of this produ ct or docum ent may be reprodu ced in any form by any m eans

with out p rior written au thor ization of Sun and its licensors, if any.

Third-p arty software, includ ing font techn ology, is copyr ighted an d licensed from Sun sup pliers.

Parts of the prod uct may be d erived from Berkeley BSD systems, licensed from the Univer sity of California. UNIX is a

registered trad emar k in the U.S. and o ther coun tries, exclusively licensed th rou gh X/ Open Com pa ny, Ltd.

Sun, Sun Microsystems, th e Sun Logo, Solaris, NFS, Sun OS, Solstice SyMON , HotJava, Solstice DiskSuite, Sun Tru nkin g,

SunVTS, OpenWind ow s, OpenBoot, Ultra Enterpr ise, Sun StorEdge, SunN et, CacheFS, SunVideo, Sun Swift and VideoPix

are trad emar ks or registered trad emar ks of Sun Microsystems, Inc. in the U.S. and other coun tries.

All SPARC tradem arks are used u nd er license and are trad emar ks or registered trad emar ks of SPARC Internation al, Inc.

in the U.S. and o ther coun tries. Prod ucts bearing SPARC tradem arks are based up on an architecture develop ed by Sun

Microsystems, Inc.

The OPEN LOOK and Sun Grap hical User Interface was d eveloped by Sun Microsystems, Inc. for its user s and licensees.

Sun acknow ledges the pion eering efforts of Xerox in researching and dev eloping th e concept of visual or grap hical user

interfaces for the compu ter ind ustr y. Sun hold s a non -exclusive license from Xerox to the Xerox Grap hical User Interface,

wh ich license also covers Suns licensees wh o implem ent OPEN LOOK GUIs and otherw ise comp ly with Suns wr itten

license agreemen ts.

U.S. Governm ent app roval required w hen exporting the prod uct.

RESTRICTED RIGHTS: Use, du p lication, or disclosu re by the U.S. Govern men t is subject to restr ictions of FAR 52.227-

14(g) (2)(6/ 87) and FAR 52.227-19(6/ 87), or DFAR 252.227-7015 (b)(6/ 95) and DFAR 227.7202-3(a).

DOC UMEN TATION ISPROVIDED "AS IS"A ND ALL EXPRESSO R IMPLIED CON DITION S, REPRESENTATION S,

AN D WARRAN TIES, INCLUDIN G ANY IMPLIED WARRAN TYOF MERCHANTABILITY, FITNESSFOR A

PARTICULAR PURPOSE OR NON -INFRING EMENT, ARE DISCLAIMED, EXCEPT TO TH E EXTENT THAT SUCH

DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.


3/698


4/698

iv Solaris S ystem Performance M anagement Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999,Revision B

sar....................................................................................................... 2-7vmstat ................................................................................................ 2-8iostat ................................................................................................ 2-9mpstat .............................................................................................. 2-10netstat ............................................................................................ 2-11nfsstat ............................................................................................ 2-12SyMON System Monitor................................................................ 2-13Other Utilities .................................................................................. 2-14memtool ............................................................................................ 2-15/usr/proc/bin............................................................................... 2-17The Process Manager ...................................................................... 2-18The SE Toolkit.................................................................................. 2-20System Perform ance Monitoring With the SE Toolkit .............. 2-21SE Toolkit Example Tools .............................................................. 2-22System Accou nting ......................................................................... 2-23Viewing Tunable Param eters ........................................................ 2-24

Setting Tunable Parameters........................................................... 2-25/etc/system ................................................................................... 2-26Check You r Progress ...................................................................... 2-27Think Beyond .................................................................................. 2-28

System Monitoring Tools ...................................................................... L2-1Tasks ................................................................................................ L2-2

Using the Tools....................................................................... L2-2Enabling Accounting .............................................................L2-3Installing the SE Toolkit ........................................................ L2-3Installing memtool................................................................ L2-4

Processes and Threads ..............................................................................3-1Relevance............................................................................................ 3-2Process Lifetime ................................................................................ 3-3forkWith exec Exam ple................................................................. 3-4Process Performance Issues............................................................. 3-5Process Lifetime Performance Data................................................ 3-6Process-Related Tun able Param eters ............................................. 3-8Mu ltith read ing ................................................................................ 3-10Ap plication Thread s ....................................................................... 3-12Process Execution ............................................................................ 3-13Thread Execution ............................................................................ 3-14

Process Thread Examples .............................................................. 3-15Perform ance Issues ......................................................................... 3-16Thread and Process Perform ance ................................................. 3-17Locking ............................................................................................. 3-18Locking Problems ........................................................................... 3-20The lockstatCommand .............................................................. 3-21Inter ru pt Levels ............................................................................... 3-22The clockRoutine .......................................................................... 3-23


5/698

Solaris System Performance Management vCopyr ight 1999Sun Microsystems, Inc. All Rights Reserved . Enterp rise Services October 1999, Revision B

Clock Tick Processing ..................................................................... 3-24Process Monitoring Using ps........................................................ 3-25Process Monitoring Using System Accounting .......................... 3-28

msacct.se................................................................................3-32pea.se ......................................................................................3-32ps-ax.se..................................................................................3-34ps-p.se ....................................................................................3-34pwatch.se................................................................................3-34

Check You r Progress ...................................................................... 3-35Think Beyond .................................................................................. 3-36

Processes Lab ........................................................................................... L3-1Tasks ................................................................................................ L3-2

Using maxusers .................................................................... L3-2Using lockstat ..................................................................... L3-3Monitoring Processes ............................................................ L3-3

CPU Scheduling .........................................................................................4-1Relevance............................................................................................ 4-2Sched uling States .............................................................................. 4-3Sched uling Classes ............................................................................ 4-4

Class Characteristics .................................................................4-5Disp atch Priorities ............................................................................. 4-6Timesharing/ Interactive Dispatch Param eter Table ................... 4-7

CPU Starva tion ..........................................................................4-8Dispatch Param eter Table Issues .................................................... 4-9Timesharing Dispatch Parameter Table (Batch)......................... 4-10The dispadminComman d ............................................................ 4-12

dispadminExample ....................................................................... 4-13The Interactive Scheduling Class.................................................. 4-14Real-Time Sched uling .................................................................... 4-15Real-Time Issues.............................................................................. 4-16Real-Time Dispatch Parameter Table........................................... 4-18The priocntlComman d .............................................................. 4-19Kernel Thread Dispatching ........................................................... 4-21

Old Thread ...............................................................................4-21New Thread .............................................................................4-22

Processor Sets .................................................................................. 4-23Dynamic Reconfigu ration Interaction ..................................4-24

The Run Qu eu e................................................................................ 4-27CPU Activity .................................................................................... 4-28CPU Perform an ce Statist ics ........................................................... 4-30Available CPU Performance Data ................................................ 4-34CPU Control and Monitoring........................................................ 4-35The mpstatComman d ................................................................... 4-36mpstatData ..................................................................................... 4-38Solaris Resource Manager.............................................................. 4-39


6/698

vi Solaris S ystem Performance M anagement Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999,Revision B

SRM Hierarchical Shares Exam ple ............................................... 4-40SRM Limit Nod e ............................................................................. 4-41CPU Monitoring Using SE Toolkit Programs............................. 4-42

cpu_meter.se.........................................................................4-42vmmonitor.se.........................................................................4-42mpvmstat.se ...........................................................................4-42


CPU Scheduling Lab .............................................................................. L4-1Tasks ................................................................................................ L4-2

CPU Dispatching.................................................................... L4-2Process Pr iorities .................................................................... L4-3The Dispatch Parameter Table ............................................. L4-6Real-Time Sched uling ........................................................... L4-7

System Caches ............................................................................................5-1

Relevance............................................................................................ 5-2What Is a Cache? ............................................................................... 5-3Hardware Caches.............................................................................. 5-4SRAM an d DRAM ............................................................................ 5-5CPU Caches........................................................................................ 5-6Cache Op eration ................................................................................ 5-7Cache Miss Processing ..................................................................... 5-8Cache Rep lacement ........................................................................... 5-9Cache Op eration Flow .................................................................... 5-10Cache Hit Rate ................................................................................. 5-11Effects of CPU Cache Misses ......................................................... 5-13

Cache Characteristics...................................................................... 5-14Virtual Address Cache ................................................................... 5-15Physical Address Cache ................................................................. 5-16Direct Mapped and Set Associative Caches................................ 5-17

Harvard Caches .......................................................................5-18Write-Through and Write-Back Caches....................................... 5-19

Write Cancellat ion ..................................................................5-19CPU Cache Snooping ..................................................................... 5-21Cache Thrashing.............................................................................. 5-23System Cache Hierarchies ............................................................. 5-25Relative Access Times .................................................................... 5-27

Cache Performance Issues ............................................................. 5-28Things You Can Tune..................................................................... 5-30Check You r Progress ...................................................................... 5-31Think Beyond .................................................................................. 5-32

Cache Lab.................................................................................................. L5-1Tasks ................................................................................................ L5-2

Cache Miss Costs .................................................................... L5-2


7/698

Solaris System Performance Management viiCopyr ight 1999Sun Microsystems, Inc. All Rights Reserved . Enterp rise Services October 1999, Revision B

Memory Tuning .........................................................................................6-1Objectives ........................................................................................... 6-1Relevance............................................................................................ 6-2Virtu al Memory ................................................................................. 6-3Typ es of Segm ents ............................................................................ 6-6Virtu al Ad dress Translat ion ............................................................ 6-7Page Descrip tor Cache ..................................................................... 6-8Virtual Address Lookup ................................................................ 6-10The Memory Free Qu eu e ............................................................... 6-11The Paging Mechan ism .................................................................. 6-12Page Daemon Scanning.................................................................. 6-13Page Scanner Processin g ................................................................ 6-14Defau lt Paging Parameters ............................................................ 6-15An Alternate Approach The Clock Algorithm ........................ 6-16maxpgio ............................................................................................ 6-18File System Caching........................................................................ 6-20

The Buffer Cache ............................................................................. 6-22The Page Cache ............................................................................... 6-24Priority Paging................................................................................. 6-25Available Paging Statistics............................................................. 6-27Available Paging Data.................................................................... 6-31Additional Paging Statistics .......................................................... 6-32Swapping.......................................................................................... 6-34What Is Not Swappable ................................................................. 6-36Swapping Priorities ........................................................................ 6-37Swap In ............................................................................................. 6-38Swap Space ...................................................................................... 6-39

tmpfs ................................................................................................ 6-41Available Swap ping Statistics ....................................................... 6-43Available Swapping Data .............................................................. 6-46Shared Librar ies .............................................................................. 6-47Memory Utilization ........................................................................ 6-49Tota l Physical Memory .................................................................. 6-50File Buffering Memory ................................................................... 6-51Kern el Memory ............................................................................... 6-52Iden tifying an Applications Memory Requ irements ................ 6-54Free Memory.................................................................................... 6-56Identifying a Mem ory Shortage .................................................... 6-57

Using the memtoolGUI.................................................................... 6-59Things You Can Tune..................................................................... 6-63Check You r Progress ...................................................................... 6-65Think Beyond .................................................................................. 6-66


8/698

viii Solaris S yst em Performance M anagement Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999,Revision B

Memory Tuning Lab .............................................................................. L6-1Tasks ................................................................................................ L6-2

Process Mem ory ..................................................................... L6-2Page Daemon Parameter s ..................................................... L6-3Scan Rates ................................................................................ L6-7

System Buses...............................................................................................7-1Relevance............................................................................................ 7-2What Is a Bus? ................................................................................... 7-3General Bus Characteristics............................................................. 7-4System Buses...................................................................................... 7-5

MBus an d XDbus ......................................................................7-6UPA and Gigaplane Bus ..........................................................7-7

Gigap lane XB Bus.............................................................................. 7-8The Gigap lan e XB Bus ...................................................................... 7-9Sun System Backp lanes Sum mary................................................ 7-10Perip heral Buses .............................................................................. 7-11

SBus Versus PCI Bus....................................................................... 7-12Enterprise Server I/ O Boards................................................7-13

PIO an d DVMA ............................................................................... 7-14PIO.............................................................................................7-15DVMA.......................................................................................7-15Which Is Being Used?.............................................................7-15

prtdiag ............................................................................................ 7-16prtdiagCPU Section ..................................................................... 7-17prtdiagMemory Section .............................................................. 7-18

Memory Interleaving..............................................................7-18prtdiag I/ O Section ...................................................................... 7-20

Bus Limits......................................................................................... 7-21Bus Ban dwidth ................................................................................ 7-22Diagnosing Bus Problems.............................................................. 7-23Avoid ing the Problem .................................................................... 7-24SBus Overload Example................................................................. 7-26Dynamic Reconfigu ration Con sidera tions .................................. 7-28Tuning Repor ts ................................................................................ 7-29Check You r Progress ...................................................................... 7-30Think Beyond .................................................................................. 7-31

System Buses Lab.................................................................................... L7-1

Tasks ................................................................................................ L7-2System Configu ration ............................................................ L7-2

I/O Tuning ...................................................................................................8-1Relevance............................................................................................ 8-2SCSI Bus Overview ........................................................................... 8-3SCSI Bus Characteristics .................................................................. 8-5SCSI Speeds........................................................................................ 8-6


9/698

Solaris System Performance Management ixCopyr ight 1999Sun Microsystems, Inc. All Rights Reserved . Enterp rise Services October 1999, Revision B

SCSI Widths ....................................................................................... 8-8SCSI Lengths ...................................................................................... 8-9SCSI Properties Summary.............................................................. 8-10SCSI Inter face Ad dressing ............................................................. 8-11SCSI Bus Addressing ...................................................................... 8-13SCSI Target Pr iorities ..................................................................... 8-14Fibre Chan nel................................................................................... 8-15Disk I/ O Time Components.......................................................... 8-18

Access to the I/ O Bus .............................................................8-18Bus Transfer Time ...................................................................8-20Seek Time .................................................................................8-20Rota tion Time ..........................................................................8-20ITR Transfer Time ...................................................................8-21Reconnection Time..................................................................8-21Interru pt Time .........................................................................8-21

Disk I/ O Time Calcu lation ............................................................ 8-22

Bus Data Transfer Timing......................................................8-23Latency Timing........................................................................8-23Disk Drive Featu res ........................................................................ 8-24Multiple Zone Recording............................................................... 8-25Drive Caching and Fast Writes ..................................................... 8-27

Fast Writes ................................................................................8-27Tagged Qu eu eing ............................................................................ 8-29

Ordered Seeks ..........................................................................8-30Mode Pages ...................................................................................... 8-31Device Properties From prtconf ................................................. 8-32Device Properties From scsiinfo ............................................... 8-34

I/ O Performan ce Planning ............................................................ 8-36Decision Support Examp le (Sequen tial I/ O) ......................8-37Transaction Processing Examp le (Random I/ O) ...............8-37

Tap e Drive Operat ions ................................................................... 8-38Storage Arrays and JBODs............................................................. 8-39Storage Array Architecture............................................................ 8-40RAID ................................................................................................. 8-41RAID-0.............................................................................................. 8-42RAID-1.............................................................................................. 8-44RAID-0+1.......................................................................................... 8-46RAID-3.............................................................................................. 8-48

RAID-5.............................................................................................. 8-50Tuning the I/ O Subsystem ............................................................ 8-52Available Tuning Data ................................................................... 8-54Available I/ O Statistics .................................................................. 8-55Available I/ O Data ......................................................................... 8-58iostat Service Time....................................................................... 8-59

siostat.se .............................................................................8-60xio.se ......................................................................................8-60


10/698

x Solaris System Performance Management Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999,Revision B

xiostat.se .............................................................................8-61iomonitor.se.........................................................................8-61iost.se ....................................................................................8-61xit.se ......................................................................................8-62disks.se..................................................................................8-62


I/O Tuning Lab ........................................................................................ L8-1Tasks ................................................................................................ L8-2

SCSI Device Characteristics .................................................. L8-2

UFS Tuning .................................................................................................9-1Relevance............................................................................................ 9-2The vnode Interface .......................................................................... 9-3The Local File System ....................................................................... 9-5Fast File System Layou t ................................................................... 9-6

Disk Label...................................................................................9-6Boot Block...................................................................................9-6Sup erblock..................................................................................9-7Cylinder Grou ps........................................................................9-7

The Directory ..................................................................................... 9-9Directory Nam e Lookup Cache (DNLC)..................................... 9-10The inodeCache ............................................................................. 9-12DNLC and inodeCache Management ........................................ 9-14The inode......................................................................................... 9-16

inode Time Stamp Updates ..................................................9-17The inodeBlock Pointer s .............................................................. 9-18

UFS Buffer Cache ............................................................................ 9-19Hard Links ....................................................................................... 9-21Symbolic Links ................................................................................ 9-22Allocating Directories and inodes ............................................... 9-23Allocation Performance.................................................................. 9-24Allocating Data Blocks ................................................................... 9-25Qu ad ra tic Rehash ............................................................................ 9-26

The Algorithm .........................................................................9-27Fragm ents ......................................................................................... 9-28Logging File Systems...................................................................... 9-29Ap plication I/ O............................................................................... 9-30

Working With the segmapCache ................................................. 9-33File System Performance Statistics ............................................... 9-34fsflush ............................................................................................ 9-37Direct I/ O......................................................................................... 9-39

Restrictions...............................................................................9-40Accessing Data Directly With mmap.............................................. 9-41Using madvise................................................................................. 9-42


11/698

Solaris System Performance Management xiCopyr ight 1999Sun Microsystems, Inc. All Rights Reserved . Enterp rise Services October 1999, Revision B

File System Per forman ce................................................................ 9-43Data-Intensive or Attribu te-Intensive App lications ..........9-43Sequential or Random Accesses ...........................................9-44

Cylinder Grou ps.............................................................................. 9-45Sequential Access Workloads........................................................ 9-47UFS File System Read Ahead ........................................................ 9-49Setting Cluster Sizes for RAID Volumes ..................................... 9-51Com mands to Set Cluster Size ...................................................... 9-53

Tuning for Sequ ential I/ O .....................................................9-54File System Write Behind ............................................................... 9-55UFS Write Throttle .......................................................................... 9-57Random Access Workloads........................................................... 9-59Tuning for Random I/ O ................................................................ 9-61Post -Creation Tuning Param eter s ................................................ 9-63

Fragment Optimization..........................................................9-63minfree ....................................................................................9-64

maxbpg ......................................................................................9-64Ap plication I/ O Perform ance Stat istics....................................... 9-65Check You r Progress ...................................................................... 9-67Think Beyond .................................................................................. 9-68

UFS Tuning Lab ...................................................................................... L9-1Tasks ................................................................................................ L9-2

File Systems............................................................................. L9-2Directory Nam e Looku p Cache ........................................... L9-3mmap an d read/write........................................................... L9-6Using madvise........................................................................ L9-8

Network Tuning .......................................................................................10-1Relevance.......................................................................................... 10-2Networks .......................................................................................... 10-3Network Ban dwidth ....................................................................... 10-4Full Duplex 100-BaseT Ethernet.................................................... 10-5

ndd Exam ples ...........................................................................10-6Forcing 100-Base T Full Duplex ............................................10-6

IP Tru nkin g ...................................................................................... 10-7Sun Trunking Software ..........................................................10-8

TCP Connections............................................................................. 10-9TCP Stack Issues............................................................................ 10-11

TCP Tuning Param eters ............................................................... 10-12Using ndd ....................................................................................... 10-14Network Hardware Performance............................................... 10-16NFS.................................................................................................. 10-18NFS Retran smission s .................................................................... 10-19NFS Server Daemon Thread s ...................................................... 10-21


12/698

xii Solaris S yst em Performance M anagement Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999,Revision B

Mon itorin g N etw ork Performance............................................. 10-23ping........................................................................................10-24spray .....................................................................................10-25snoop .....................................................................................10-26

Isolating Problems ........................................................................ 10-27Tuning Repor ts .............................................................................. 10-28Network Monitoring Using SE Toolkit Programs ................... 10-29

net.se ....................................................................................10-29netstatx.se .........................................................................10-29nx.se ......................................................................................10-29netmonitor.se.....................................................................10-30nfsmonitor.se.....................................................................10-30tcp_monitor.se...................................................................10-30

Check You r Progress .................................................................... 10-31Think Beyond ................................................................................ 10-32

Performance Tuning Summary .............................................................11-1

Relevance.......................................................................................... 11-2Some General Guidelines............................................................... 11-3What You Can Do ........................................................................... 11-4Ap plication Source Cod e Tun ing ................................................. 11-5Compiler Optimization .................................................................. 11-6Ap plication Executable Tun ing Tips ............................................ 11-7Performance Bottlenecks................................................................ 11-8Memory Bottlen ecks ..................................................................... 11-10Memory Bottleneck Solutions ..................................................... 11-11I/ O Bottlen ecks ............................................................................. 11-13I/ O Bottlen eck Solutions ............................................................. 11-15

CPU Bottlen ecks ............................................................................ 11-17CPU Bottleneck Solu tion s ............................................................ 11-19Ten Tuning Tips ............................................................................ 11-21Case Study...................................................................................... 11-23Check You r Progress .................................................................... 11-46Think Beyond ................................................................................ 11-47

Performance Tuning Summary Lab .................................................. L11-1Tasks .............................................................................................. L11-2

vmstat Inform ation .............................................................L11-2sarAn alysis I ....................................................................... L11-3

sarAn alysis II...................................................................... L11-6sarAn alysis III ....................................................................L11-9

Interface Card Properties .........................................................................A-1Ad ditional Resou rces ...................................................................... A-2SBus Implementations and Capabilities....................................... A-3Typical and Peak Bandwidth of SBus Devices ............................ A-4


13/698

Solaris System Performance Management xiiiCopyr ight 1999Sun Microsystems, Inc. All Rights Reserved . Enterp rise Services October 1999, Revision B

Installing and Configuring the SyMON System Monitor................ B-1Ad dition al Resou rces ...................................................................... B-2SyMON System Monitor ................................................................. B-3Sup ported Server Platforms ........................................................... B-4Up grad ing the SyMON System Monitor ...................................... B-5Installing the SyMON System Mon itor ........................................ B-6Installing the SyMON Packages .................................................... B-7

Env ironment Variables............................................................ B-7Configuring th e SyMON System Mon itor ................................... B-8

Configuring the Monitored Server System .......................... B-8Run ning the SyMON User In terface .....................................B-8Configuring the Event H andler System ............................... B-9

Setting Up SNMP With Solstice Domain (Sun Net)Manager ........................................................................................ B-10

Directin g the SNMP Trap s.................................................... B-10Writing or Mod ifying Event Rules to Send SNMP

Traps ..................................................................................... B-11How to Stop or Remove SyMON From the System .................. B-12Configu ration Issu es ...................................................................... B-13

Accounting .................................................................................................C-1Overview ........................................................................................... C-2

What Is Accounting? ...............................................................C-2What Is Accou nting Used For? ..............................................C-2

Types of Accounting........................................................................ C-3Connection Accounting...........................................................C-3Process Accounting..................................................................C-3Disk Accou nting .......................................................................C-4

Chargin g ....................................................................................C-4How Does Accounting Work? ....................................................... C-5

Location of Files .......................................................................C-5Typ es of Files ............................................................................C-5What Hap pens at Boot Time ..................................................C-5Programs That Are Run ..........................................................C-6Location of ASCII Report Files...............................................C-7

Starting and Stopping Account ing ................................................ C-8Genera ting the Data and Reports ................................................ C-10

Raw Data .................................................................................C-10rprtMMDD Daily Report File............................................C-12Generation of the rprtMMDD File....................................C-12The runacct Program ...........................................................C-17runacct States ........................................................................C-18Periodic Repor ts .....................................................................C-20Looking at the pacct File With acctcom ...........................C-21acctcomOp tions ....................................................................C-22

Genera ting Custom Analyses....................................................... C-24


14/698

xiv Solaris S yst em Performance M anagement Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999,Revision B

Writing Your Ow n Program s ...............................................C-24Raw Data Formats..................................................................C-24/var/adm/wtmpFile ..............................................................C-25/var/adm/pacctFile ............................................................C-26/var/adm/acct/nite/disktacctFile .............................C-27

Summary of Accoun ting Program s an d Files ............................ C-28

The Cache File System .............................................................................D-1Introd uction ...................................................................................... D-2

Function .....................................................................................D-2Characteristics ..........................................................................D-2Terminology ..............................................................................D-2Benefits ......................................................................................D-3Limitations ................................................................................D-4Required Software ...................................................................D-4

Settin g Up CacheFS ......................................................................... D-5Creating a Cache ......................................................................D-5

Disp laying Inform ation ...........................................................D-5Deletin g a Cache ............................................................................... D-6Mounting With CacheFS................................................................. D-7

Using an Alread y Mounted NFS File Systems ....................D-7Using /etc/vfstab.................................................................D-7Using the Automounter ..........................................................D-7

cachefsOp eration .......................................................................... D-8fsck............................................................................................D-8Writes to Cached Data.............................................................D-8

Managing cachefs.......................................................................... D-9cachefsstat ............................................................................D-9

cachefslog ..............................................................................D-9cachefswssize......................................................................D-10cachefspack ..........................................................................D-10

CacheFS and CD-ROMs ................................................................ D-11Exercise: Using CacheFS ............................................................... D-12

Preparation ..............................................................................D-12Tasks ........................................................................................D-13

IPC Tuneables............................................................................................ E-1Ad ditional Resources ...................................................................... E-2Overview ........................................................................................... E-3

How to Determ ine Curren t IPC Values ................................ E-3How to Change IPC Parameters ............................................ E-4Tuneable Parameter Limits ..................................................... E-5

Shared Memory ................................................................................ E-6Semap hores ....................................................................................... E-8Message Qu eu es ............................................................................. E-11


15/698

Solaris System Performance Management xvCopyr ight 1999Sun Microsystems, Inc. All Rights Reserved . Enterp rise Services October 1999, Revision B

Performance Monitoring Tools .............................................................. F-1Ad dition al Resources ...................................................................... F-2sar...................................................................................................... F-3

Disp lay File Access Op era tion Statistics ............................... F-3Disp lay Buffer Activity Statistics ........................................... F-3Display System Call Statistics ................................................ F-4Display Disk Activity Statistics.............................................. F-6Display Pageou t and Memory Freeing Statistics

(in Averages).......................................................................... F-7Display Kernel Memory Allocation (KMA) Statistics ........ F-8Display Interp rocess Communication Statistics.................. F-9Disp lay Pagein Stat istics ......................................................... F-9Display CPU and Swap Queue Statistics ........................... F-10Display Available Memory Pages and Swap Space

Blocks .................................................................................... F-11Disp lay CPU Utilization (the Defau lt) ................................ F-11

Display Process, Inode, File, and Shared Memory TableSizes....................................................................................... F-12Display Swapping and Switching Statistics....................... F-12Disp lay Monitor Term inal Device Stat istics ...................... F-13Disp lay All Stat istics .............................................................. F-14Other Op tions ......................................................................... F-15

vmstat ............................................................................................. F-16Display Virtual Memory Activity Sum marized at Some

Op tional, User-Supplied Interval (in Second s)............... F-16Disp lay Cache Flush ing Statistics ........................................ F-19Disp lay Interrupts Per Device.............................................. F-19

Display System Event Information, Coun ted SinceBoot ....................................................................................... F-20Display Swapp ing Activity Sum marized at Some

Op tional, User-Supplied Interval (in Second s)............... F-21iostat ............................................................................................. F-22

Disp lay CPU Statistics ........................................................... F-22Display Disk Data Transferred and Service Times ........... F-22Disp lay Disk Op erations Statistics ...................................... F-23Disp lay Device Error Sum mary Stat istics .......................... F-23Disp lay All Device Errors ..................................................... F-24Display Counts Rather Than Rates, Where Possible ........ F-24

Report on On ly the First nDisks ......................................... F-24Display Data Throu ghp ut in Mbytes/ sec Rather Than

Kbytes/ sec ........................................................................... F-25Display Device Nam es as Mount Points or Device

Ad dresses ............................................................................. F-25Display Per-Par tition Statistics as Well as Per-Device

Statistics ................................................................................ F-25Disp lay Per-Partition Statistics On ly................................... F-26


16/698

xvi Solaris S yst em Performance M anagement Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999,Revision B

Display Characters Read and Written to Terminals ......... F-26Display Extend ed Disk Statistics in Tabu lar Form ........... F-27Break Down Service Time, svc_t, Into Wait and Active

Times an d Put Device Nam e at the End of the Line ...... F-28mpstat ............................................................................................. F-29netstat ........................................................................................... F-31

Show the State of All Sockets ............................................... F-31Show All Interfaces Under Dynamic Host Configuration

Protocol (DHCP) Control................................................... F-34Show Interface Mu lticast Group Membersh ips................. F-34Show State of All TCP/ IP Interfaces................................... F-35Show Detailed Kernel Memory Allocation Information .. F-36Show STREAMS Sta tistics .................................................... F-37Show Multicast Routing Tables ........................................... F-37Show Mu lticast Routing Statistics ....................................... F-38Show Network Addresses as Nu mbers, Not Nam es........ F-38

Show the Address Resolution Protocol (ARP) Tables...... F-39Show the Routing Tables ...................................................... F-39Show Per Protocol Statistics ................................................. F-41Show Extra Socket and Rout ing Table Information.......... F-42

nfsstat ........................................................................................... F-44Show Client Information On ly............................................. F-44Show NFS Mount Options.................................................... F-46Show NFS Inform ation; Both Client and Server................ F-47Show RPC Inform ation ......................................................... F-48Show Server Information only ............................................. F-49


17/698

xviiCopyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enterp rise Services October 1999,Revision B

About This Course

Course Goal

This course provides an introduction to performance tuning for larger

Sun environm ents. It covers most of the major areas of systemperformance, concentrating on ph ysical memory and inpu t/ outpu t

(I/ O) management.

The Sun environment is qu ite complex. There are thousand s of

possible hardw are configurations, and tens of thousand s of software

configurations. Because it is not p ossible to give cookbook-style

tuning information, this course examines how the components operate

and interoperate. This und erstanding can be used to determine what

kinds of problems can arise, how to identify them, and how to

alleviate them.


18/698

x viii S olaris S yst em Performan ce Man agemen t

Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B

CourseOverview

An overview of the operation of various system hard ware and

software components is given first, then specific areas are examined.

This provides an understanding of what effect a certain tuning changewill have and whether it is necessary.

This course is intended for system administrators and system

designers with a background in Sun SPARC system administration.


19/698

About This Course

Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B

CourseMap

The following course map enables you to see the general topics and

the modules for that topic area in reference to the course goal:

Overview

Introduction toPerformanceManagement

Tools

SystemMonitoring Tools

CPU

Processes andThreads

CPU Scheduling

Memory

System Caches Memory Tuning

I/O

SystemBuses

I/O Tuning UFS Tuning

Networks

Network Tuning

Summary

PerformanceTuning

Summary


20/698

xx Solaris S ystem Performance M anagement


Module-by-ModuleOverview

This course contains the following modules:

q

Modu le 1 Introdu ction to Performan ce Management

This modu le introdu ces the concepts of tuning, tuning tasks, and

terminology, and the steps involved in tuning a system. A list of

tuning references is also provided.

q Module 2 System Monitoring Tools

The various performance monitoring tools available with the

Solaris Operating Environmen t (Solaris), the information th at

they provide, and how to inspect and set system tun ing

param eters are described in th is modu le.


21/698

About This Course


q Module 3 Processes and Threads

This module provides a description of the behavior of processes

and threads in the Solaris environment. It covers their creation,

execution, and related tuning issues, as well as system timing and

the clock thread.

q Module 4 CPU Scheduling

How to control work running on the systems central processing

un its (CPUs), d ispatch p riorities, do real-time p rocessing, and use

processor sets are all discussed in this module.

q Module 5 System Caches

This module describes the generic operation of caches and the

specific operation of the hardware caches. Since many system

componen ts use caches or op erate like one, un derstand ing cachesis fun dam ental to being able to manage system p erformance.

q Modu le 6 Memory Tun ing

The operation of the virtual and real memory subsystems is

covered in this module. Because it is perhaps the most important

componen t of system p erformance, memory operation is

discussed in some detail here.


22/698

xxii Solaris S ystem Perform ance M anagem ent


q Module 7 System Buses

This mod ule shows how buses operate, wh at kinds of buses there

are in the system, and what bus characteristics are important to

tuning.

q Module 8 I/ O Tuning

In this mod ule, the characteristics of the small compu ter system

interface (SCSI) bus and d isk drives and their interactions are

described. It covers redundant array of inexpensive disks (RAID)

levels and storage arrays, and looks at the cost of I/ O operations

to enable the system administrator to quickly identify and locate

the source of an I/ O problem.

q Module 9 UFS Tun ing

This module explains the structure and operation of the Solaris

UNIX file system (UFS). It discusses the various UFS disk

structures, caches, and allocation algorithm s. A description of the

various UFS configuration and tun ing param eters is also provided.


23/698

About This Course


q Modu le 10 Network Tun ing

The tuning required for a network, both the physical network an d

the network applications, is covered in this module. It discusses

tun ing the Transmission Con trol Protocol/ Internet Protocol

(TCP/ IP) stack and NFS.

q Module 11 Performance Tuning Summary

This module summarizes the material covered in this course. It

offers examp les of poor performance and their causes. It also

provides some general guidelines for tuning, a brief discussion of

application optimization, and suggestions for dealing with

common system performance bottlenecks.


24/698

xxiv Solaris S ystem Perform ance M anagem ent


Appendices

q Appendix A Interface Card Properties

A sum mary of the characteristics of many of the I/ O interfacecards available for the SPARC systems is given in th is app end ix.

q Appendix B Installing and Configuring SyMON System

Monitor

This appendix discusses how to install, configure, and operate the

Solaris SyMONTM system monitor.

q App end ix C Accoun ting

This appen dix d iscusses the configuration and opera tion of Solaris

system accounting.

q Appen dix D The Cache File System

The configuration, operation and use of CacheFS on both the

client and the server is discussed in this appendix.


25/698

About This Course


q Appen dix E IPC Tuneables

While not a d irect part of the course, the setting of the Interprocess

Commun ication (IPC) shared memory and semaph ore parameters

is important. This appendix explains all of the IPC tunable

parameters.

q Appen dix F Performance Monitoring Tools

This appendix is to be used as a reference for the main

per formance monitoring tools that are par t of the Solaris operating

system. Sample output and descriptions of the fields are given for

the following comman ds: sar, vmstat, iostat, mpstat, netstat,

an d nfsstat.


26/698

xxvi Solaris S ystem Perform ance M anagem ent


CourseObjectives

Upon completion of this course, you should be able to:

q

Describe the pu rpose and goals of tuningq Describe the common characteristics of system hardware and

software comp onents

q Understand how the major system components operate

q Und erstand the meanings of tuning report elements and h ow they

relate to system performance

q Identify the usual tuning parameters and understand when and

how to set them

q Describe the different types of performance graphs

q Determine whether a problem exists in system hard ware

configurations

q Identify main memory problems and their solutions

q Identify I/ O hardware and file system performance problems and

their solutions

q Identify network p erformance problems and their solutions

q Und erstand CPU scheduling and the functions of threads in theSolaris operating system


27/698

About This Course


Skills Gained by Module

The skills for Solaris System Performance Management are shown in

column 1 of the following m atrix. The black boxes ind icate the main

coverage for a top ic; the gray boxes ind icate the topic is brieflydiscussed.

Module

Skills Gained 1 2 3 4 5 6 7 8 9 10 11

Describe the pu rpose and goals of tuning

Describe the common characteristics of systemhardware and software comp onents

Und erstand how the major system comp onentsoperate

Und erstand the meanings of tun ing reportelements and how they relate to systemperformance

Identify the usu al tuning param eters andund erstand w hen and h ow to set them

Describe the d ifferent typ es of performancegraphs

Determine w hether a problem exists in systemhardware configurations

Identify main memory problems and theirsolutions

Identify netw ork performan ce problems and th eirsolutions

Understand CPU scheduling and the functions ofthread s in the Solaris operating system


28/698

x xv iii S olaris S yst em Performan ce Man agemen t


Guidelines for ModulePacing

The following table provides a rough estimate of pacing for this

course:

Module Day 1 Day 2 Day 3 Day 4 Day 5

"About This Course" A.M.

"Introdu ction to PerformanceManagement"

A.M.

"System Monitor ing Tools" P.M.

Processes an d Th read s P.M.

CPU Scheduling A.M.

System Caches P.M.

Memory Tuning A.M.

System Buses P.M.

I/ O Tuning A.M.

UFS Tuning P.M.

Network Tuning A.M.

Performance Tuning Summary P.M.


29/698

About This Course


Topics Not Covered

This course does not cover the topics shown on the above overhead .

Many of these topics are covered in other courses offered by Sun

Educational Services.

q Capacity p lanning Covered in ES-360: Planning and Managing

A7000 Intelligent Storage Servers

q General system administration Covered in SA-237: Solaris 7

System Administration I, SA-287: Solaris 7 System A dministration II,

and SA-350: Solaris 2.x Server Administration

q Netw ork ad ministration an d man agement Covered in SA-387:

Solaris TCP/IP Network Administration

q General Solaris problem resolution Covered in ST-350: SunSystems Fault Analysis Workshop


30/698

xxx Solaris S ystem Perform ance M anagem ent


q Data center operat ions Covered in RS-350: Operating Client-Server

Systems

q Disk storage managem ent Covered in SA-347: Volume Manager

With StorEdge A5000

Refer to the Sun Edu cational Services catalog for specific information

and registration.


31/698

About This Course


How Prepared AreYou?

To be sure you are p repared to take this course, can you answer yes to

the questions shown on the above overhead?

q If you have had Solaris system administration training or

experience, you will be able to install software, edit cron tables,

start daemons, and perform the lab exercises.

q If you understand the function of the kernel, you will be able to

load kernel modu les and mod ify kernel tunable parameters.

q If you have basic knowledge of Sun server platforms, you will

un derstand the role of the CPU, memory, and buses and their

relationships to system performance.

q If you h ave an u nd erstand ing of system buses, peripheral buses,types of I/ O d evices, and the UN IX file system, you w ill be able to

recognize inappropriate configurations and tune the I/ O

subsystem properly for a given application.

q If you u nd erstand network p rotocols, you w ill be able to tune

them in order to maximize the network bandwidth.


32/698

x xx ii S olaris S yst em Performan ce Man agemen t


Introductions

Now that you h ave been introdu ced to the course, introdu ce yourself

to each other an d the instructor, add ressing the items shown on the

above overhead.


33/698

About This Course


How toUsetheCourseMaterials

To enable you to su cceed in this course, these course materials emp loy

a learning mod el that is comp osed of the following components:

q Course map An overview of the course content ap pears in the

"About This Course" module so you can see how each module fits

into the overall course goal.

q Objectives - What you should be able to accomplish after

completing this module is listed here.

q Relevance This section, which appears in every module,

provides scenarios or questions that introdu ce you to the

information contained in the mod ule and p rovoke you to think

about h ow the m odu le content relates to optimizing system

performance.

q Additional resources Where you can look for more detailed

information, or information on configurations, options, or other

capabilities on the topics covered in the module or appendix.


34/698

x xx iv S olaris S yst em Performan ce Man agemen t


q Overhead image Reduced overhead images for the course are

included in the course materials to help you easily follow where

the instructor is at any point in time. Overheads do n ot app ear on

every page.

q Lecture The instru ctor will present information specific to the

topic of the module. This information will help you learn the

knowledge and skills necessary to succeed with the exercises.

q Exercise Lab exercises will give you the op por tun ity to practice

your skills and apply the concepts presented in the lecture.

q Check your progress Modu le objectives are restated , sometimes

in question format, so that before moving on to the next mod ule

you are sure that you can accomplish the objectives of the current

module.

q Think beyond Thought-provoking questions are posed to helpyou app ly the content of the module or pred ict the content in the

next module.


35/698

About This Course


Course Icons and Typographical Conventions

The following icons and typographical conventions are used in this

course to represent various training elements and alternative learning

resources.

Icons

Additional resources Indicates additional reference materials are

available.

Discussion Indicates a small-group or class discussion on th e current

topic is recomm ended at this time.

Note Additional imp ortant, reinforcing, interesting, or sp ecial

information.

!Caution A potential hazard to data or machinery.

Warning Anything that p oses personal danger or irreversible

dam age to data or the operating system.


36/698

x xx vi S olaris S yst em Performan ce Man agemen t


Typographical Conventions

Courier is used for the names of commands, files, and directories, as

well as on-screen compu ter outp ut. For examp le:

Use ls -al to list all files.system% You have mail.

Courier bold is used for characters and nu mbers that you type. For

example:

system% su

Password:

Courier italic is used for variables and command-line

placeholders tha t are rep laced w ith a real nam e or value. For example:

To d elete a file, type rm filename.

Palatino italics is used for book titles, new word s or terms, or word s

that are emp hasized. For example:

Read Chapter 6 in Users Guide.

These are called class options.

You mustbe root to do this.


37/698

1-1Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enterp rise Services October 1999,Revision B

Introduction toPerformanceManagement 1

Objectives

Upon completion of this mod ule, you should be able to:

q Explain wh at performance tun ing really means

q Characterize the type of work performed on a system to m ake

meaningful measurements

q Explain p erformance measuremen t terminology

q Describe a typical performance tuning task


38/698

1

1-2 Solaris S ystem Perform ance M anagem ent


Relevance

Discussion The following questions are relevant to un derstand ing

the content of this mod ule:

q Why tun e a system?

q How do you know that a system requires tuning?

q How d o you know what to tune?

Additional Resources

Additional resources The following references can provide

add itional d etails on the top ics d iscussed in this mod ule:

q Cockcroft, Adrian, and Richard Pettit. 1998. Sun Performance and

Tuning. Palo Alto: Sun Microsystems Press/ Prentice Hall.

q Wong, Brian L. 1997. Configuration and Capacity Planning for Solaris

Servers. Palo Alto: Sun Microsystems Press/ Prentice Hall.

q Benner, Alan F. 1996. Fibre Channel. McGraw-Hill.

q Catanzaro, Ben. 1994. Multiprocessor System Architectures. SunSoft

Press/ Prentice Hall.

q Cervone, Frank. 1998. Solaris Performance A dministration. McGraw-

Hill.

q Goodheart, Berny, and James Cox. 1994. The Magic Garden

Explained. Prentice-Hall.

q Graham, John. 1995. Solaris 2.x Internals. McGraw -Hill.

q Gunther, Neil. 1998. The Practical Performance Analyst.

McGraw-Hill.

q Raj Jain , Raj. 1991. The Art of Computer System Performance Analysis.John Wiley & Sons, Inc.

q Ridge, Peter M., and David Deming. 1995. The Book of SCSI. N o

Starch Press.

q Solaris 2 Security, Performance and Accounting in the Solaris System

Administration AnswerBook.


39/698

1

Introduction to Performance Management


q SMCC NFS Server Performance Tuning Guide in the Hardware

AnswerBook on the SMCC Supplements CD-ROM.

q The Sun web site http://www.sun.com/sun-on-

net/performance.html and related pages.

q Man p ages for the various comm and s (kstat, ps, lockstat,prtconf, prtdiag, sysdef, adb, ndd, fstyp, mkfs, newfs,

tunefs) and performance tools (sar, vmstat, iostat, mpstat,

nfsstat ,and netstat).


40/698

1



What Is Tuning?

As shown in the overhead image above, tun ing means many things. In

general, however, it means making the most efficient use of the

resources that the system has for the workload that runs on it.

Tuning is aimed at two primary activities:

q Eliminating un necessary work performed by the system

q Taking advantage of system options to tailor the system to its

workload

Almost every tuning activity can be placed into one of these two

categories.


41/698

1



BasicTuning Procedure

Basic tun ing p rocedures are quite simple: locate a bottleneck and

remove it. This process is repeated un til performan ce is satisfactory,

however that is measured.

Note that you will never be finished tun ing: there will always be a

bottleneck. If there were none, infinite work could be completed in

zero time. Tuning provides a compromise between cost, the adding of

extra hardware, and performance, the ability to get the systems work

done efficiently.

Before beginning a tuning project, you should have a goal in mind:

what is satisfactory p erformance. A realistic goal is important:

expectations can be set that cannot be met, causing even the most

successful tun ing p roject to be perceived as a failure.


42/698

1



Conceptual Model of Performance

There are many factors that make a system p erform the way it does.

These include:

q The workload itself. The workload may be I/ O bound, CPU

bound , or netw ork intensive, consisting of short jobs; long jobs;

simple queries; long, complex queries; simple text ed iting;

complex computer-aided design (CAD) simulations; or any

combination of these. It may change during the d ay or w eek.

q A rand om comp onent to the work; that is, how man y queries are

executed, how m any records are looked at, how many u sers are

supported , how much email is generated, how big files are, wh at

the file locations are, and so on. The exact same am oun t and type

of work is rarely d one repeatedly on a system.

q Changes to the system itself. These include new hardware

components, new network access, new software, new algorithms

from patches, or different media.


43/698

1



All of these items, and more, work together to make the system a very

dynamic entity, constantly changing and acting differently over time.

When tu ning, you need as m uch information as possible to make

correct decisions about the systems current and future behavior, and

where efficiencies can be obta ined.


44/698

1



WheretoTuneFirst

Take a mom ent to prioritize the areas in the overhead image for tuning

purposes from 1 (most improvement possible) to 4 (least

improvement).


45/698

1



WheretoTune?

As shown on p age -8, the closer to where the work is performed , the

more effect your tu ning w ill have. As system administrators, access to

the first tw o levels is difficult, so the tendency is to concentrate onwhat is accessible.

Tuning at the lower levels is usually an attempt to make the system

sup port the ap plications as efficiently as possible. This usu ally

involves making trad e-offs. Changes tha t benefit on e type of work

may have no effect on, or could hinder, another type of work. If you

tune the system specifically for a particular typ e of work, you must be

prepared to accept these trade-offs.

You a lso need to und erstand the characteristics of the app lications to

know how they use the system before you can tune the system

properly to support them.


46/698

1

1-10 S olaris S yst em Performan ce Man agemen t


Performance Measurement Terminology

The terminology used in the tuning literature is:

q

Bandwidth

w The peak capacity that cannot be exceeded

w A measurement that ignores overhead

w The best (perfect) case nu mbers

w Someth ing that is never achieved in p ractice; there are always

some inefficiency and overhead


47/698

1



q Throughput

w How mu ch work gets done in a specified time interval

w What you really get; that is, how mu ch of the band wid th is

actually used

w A measuremen t that is dep enden t on many factors, including

hardw are, software, human, and random

w A number that is usually impossible to calculate, but can be

approximated

w Maximum throughput if you are measuring how much work

can be d one at 100 percent u tilization

q Response time

w How long the user or requester waits for a request to complete

w Elapsed time for an operation to complete

w Usually the same thing as latency

q Service t ime

w Actual request processing time

w How long a request takes to perform, ignoring any queu eing

delays

w Respon se time, wh ich is the same as service time wh en there is

no qu eueing

q Utilization

w How mu ch of a resource, either aggregate or individual, was

used to do the measured w ork

w The amoun t of throughpu t accomplished by a resource

w A percentage usu ally expressed as a bu sy percentage


48/698

1

1-12 S olaris S yst em Performan ce Man agemen t


Performance Graphs

The three graphs on this page show the most comm on app earance of

data obtained from performance measuremen ts.

The first two graph s indicate a problem, bu t not its source. You can see

that the work p erformed , measured either as throughp ut or response

time, improves to a certain level, and then begins to drop off. The

drop-off may occur quickly or slowly, but th e system is no longer able

to efficiently perform more work. A resource constraint has been

reached.

The third g raph , a utilization grap h, shows how mu ch capacity is

available for a given resource. When fully used, the resource is nolonger available and requesters have to wait for it.

When there is a d rop-off, as measured by the first two typ es of graph ,

there w ill be a resource that is fu lly u tilized, as measured by the third

grap h. Locating th is resource and increasing its availability is the goal

of tuning.

un [1999] solaris system performance management sa-400 student guide

Documents