tips for performance bottlenecks
TRANSCRIPT
-
8/2/2019 Tips for Performance Bottlenecks
1/25
Aboutthisdocument
Thisdocumentdescribeshowtocheckforresourcebottlenecksandidentifytheprocessesthattaxthem.Resourcesonasystemincludememory,CPU,andInput/Output(I/O).Thisdocumentcoversbottlenecksacrossanentiresystem.Thisdocumentdoesnotaddressthebottlenecksofaparticularapplicationorgeneralnetworkproblems.Thefollowingcommandsaredescribed:
*vmstat*svmon*ps*tprof*iostat*netpmon*filemon
NOTE:PAIDE/6000mustbeinstalledinordertousetprof,svmon,netpmon,andfilemon.Tocheckifthisisinstalled,runthefollowingcommand:
lslpp-lperfagent.tools.
IfyouareatAIXVersion4.3.0orhigher,PAIDE/6000canbefoundontheAIXBaseOperatingSystemmedia.Otherwise,toorderPAIDE/6000,contact
yourlocalIBMrepresentative.
Thisfaxalsomakesreferencetothevmtuneandschedtunecommands.Thesecommandsandtheirsourcearefoundinthe/usr/samples/kerneldirectory.Theyareinstalledwiththebos.adt.samplesfileset.
Relatedinformation
ConsultLinePerformanceAnalysis-TheAIXSupportFamilyoffersasystemanalysiswithtuningrecommendations.FormoreinformationcalltheIBMAIXSupportCenter.
PerformanceTuningGuide(SC23-2365)-ThisIBMpublicationcovers
performancemonitoringandtuningofAIXsystems.ContactyourlocalIBMrepresentativetoorder.
Fordetailedsystemusageonaperprocessbasis,afreeutlitycalledUTLDcanbeobtainedbyanonymousftpfromftp.software.ibm.cominthe/aix/tools/perftools/utlddirectory.FormoreinformationseetheREADMEfile/usr/lpp/utldafterinstallationoftheutld.objfileset.
------------------------------------------------------------------------
Memorybottlenecks
Thefollowingsectiondescribesmemorybottlenecksolutionswiththe
followingcommands:vmstat,svmon,ps.
1.vmstat
Runthefollowingcommand:
vmstat1
NOTE:Systemmayslowdownwhenpiandpoareconsistentlynon-zero.
-
8/2/2019 Tips for Performance Bottlenecks
2/25
pinumberofpagespersecondpagedinfrompagingspaceponumberofpagespersecondpagedouttopagingspace
WhenprocessesonthesystemrequiremorepagesofmemorythanareavailableinRAM,workingpagesmaybepagedouttopagingspaceandthenpagedinwhentheyareneededagain.AccessingapagefrompagingspaceisconsiderablyslowerthanaccessingapagedirectlyfromRAM.Forthisreason,constantpagingactivitycancausesystemperformancedegradation.
NOTE:Memoryisover-committedwhenthefr:srratioishigh.
frnumberofpagesthatmustbefreedtoreplenishthefreelistortoaccommodateanactiveprocesssrnumberofpagesthatmustbeexaminedinordertofreefrnumberofpages
Anfr:srratioof1:4meansforeveryonepagefreed,fourpagesmustbeexamined.Itisdifficulttodetermineamemoryconstraintbasedon
thisratioaloneandwhatconstitutesahighratioisworkload/applicationdependent.
Thesystemconsidersitselftobethrashingwhenpo*SYS>frwhereSYSisasystemparameterviewedwiththeschedtunecommand.Thedefaultvalueis0ifasystemhas128MBormorewhichmeansthatmemoryloadcontrolisdisabled.Otherwise,thedefaultis6.Thrashingoccurswhenthesystemspendsmoretimepagingthanperformingwork.Whenthisoccurs,selectedprocessesmaybesuspendedtemporarily,andthesystemmaybenoticeablyslower.
2.svmon
Asroot,runthefollowingcommand:
#svmon-Pau10|more
Sampleoutput:
PidCommandInusePinPgspace13794dtwm16031449Pid:13794Command:dtwmSegidTypeDescriptionInusePinPgspaceAddressRangeb23pers/dev/hd2:248492000..114a5pers/dev/hd2:248420000..2
6179worklibdata1310980..891280aworksharedlibrarytext11010100..65535181workprivate28713410..310:65277..6553557d5perscode,/dev/hd2:6172282000..135
Thiscommandliststhetoptenmemoryusingprocessesandgivesareportabouteachone.Ineachprocessreport,lookwhereType=workandDescription=private.Checkhowmany4K(4096byte)pagesareusedunderthePgspacecolumn.Thisistheminimumnumberofworkingpagesthissegmentisusinginallofvirtualmemory.APgspacenumber
-
8/2/2019 Tips for Performance Bottlenecks
3/25
thatgrows,butneverdecreases,mayindicateamemoryleak.Memoryleaksoccurwhenanapplicationfailstodeallocatememory.
341*4096=1,396,736or1.4MBofvirtualmemory
3.ps
Runthefollowingcommand:
psgv|head-n1;psgv|egrep-v"RSS"|sort+6b-7-n-r
sizeamountofmemoryinKBallocatedfrompagespaceforthememorysegmentofType=workandDescription=privatefortheprocessaswouldbeindicatedbysvmonRSSamountofmemoryinKBcurrentlyinuse(inRAM)forthememorysegmentofType=workandDescription=privateplusthememorysegment(s)ofType=persandDescription=codefortheprocessaswouldbeindicatedbysvmontrsamountofmemory,inKB,currentlyinuse(inRAM)forthememorysegment(s)ofType=persandDescription=codefortheprocessaswouldbeindicatedbysvmon
%memRSSvaluedividedbythetotalamountofsystemRAMinKBmultipliedby100
------------------------------------------------------------------------
CPUbottlenecks
ThefollowingsectiondescribesCPUbottlenecksolutionsusingthefollowingcommands:vmstat,tprof,ps.
1.vmstat
Runthefollowingcommand:
vmstat1
NOTE:Systemmayslowdownwhenprocesseswaitontherunqueue.
idpercentageoftimetheCPUisidlernumberofthreadsontherunqueue
Iftheidvalueisconsistently0%,theCPUisbeingused100%ofthetime.
Looknextatthercolumntoseehowmanythreadsareplacedontherunqueuepersecond.Thehigherthenumberofthreadsforcedtowaitontherunqueue,themoresystemperformancewillsuffer.
2.tprof
TofindouthowmuchCPUtimeaprocessisusing,runthefollowingcommandasroot:
-
8/2/2019 Tips for Performance Bottlenecks
4/25
#tprof-xsleep30
Thisreturnsin30secondsandcreatesafileinthecurrentdirectorycalled__prof.all.
In30seconds,theCPUischeckedapproximately3000times.TheTotalcolumnisthenumberoftimesaprocesswasfoundintheCPU.Ifoneprocesshas1500intheTotalcolumn,thisprocesshastaken1500/3000orhalfoftheCPUtime.ThetprofoutputexplainsexactlywhatprocessestheCPUhasbeenrunning.ThewaitprocessrunswhennootherprocessesrequiretheCPUandaccountsfortheamountofidletimeonthesystem.
3.netpmon
TofindouthowmuchCPUtimeaprocessisusing,andhowmuchofthattimeisspentexecutingnetwork-relatedcode,runthefollowingcommandasroot:
#netpmon-o/tmp/netpmon.out-Ocpu-v;sleep30;trcstop
Thisreturnsin30secondsandcreatesafileinthe/tmpdirectorycallednetpmon.out.
TheCPUTimeindicatesthetotalamountofCPUtimefortheprocess.%CPUisthepercentageofCPUusagefortheprocess,andNetworkCPU%isthepercentageoftotaltimethattheprocessspentexecutingnetwork-relatedcode.
4.ps
Runthefollowingcommands:
ps-ef|head-n1ps-ef|egrep-v"UID|0:00|\0\"|sort+3b-4-n-r
ChecktheCcolumnforaprocesspenaltyforrecentCPUusage.The
maximumvalueforthiscolumnis120.
ps-e|head-n1ps-e|egrep-v"TIME|0:"|sort+2b-3-n-r
ChecktheTimecolumnforprocessaccumulatedCPUtime.
psgupsgu|egrep-v"CPU|kproc"|sort+2b-3-n-r
Checkthe%CPUcolumnforprocessCPUdependency.ThepercentCPUisthetotalCPUtimedividedbythethetotalelapsedtimesincetheprocesswasstarted.
------------------------------------------------------------------------
I/0bottlenecks
Thissectiondescribesbottlenecksolutionsusingthefollowingcommands:iostat,filemon.
1.iostat
-
8/2/2019 Tips for Performance Bottlenecks
5/25
NOTE:Highiowaitwillcauseslowerperformance.
Runthefollowingcommand:
iostat5
%iowaitpercentageoftimetheCPUisidlewhilewaitingonlocalI/O%idlepercentageoftimetheCPUisidlewhilenotwaitingonlocalI/O
ThetimeisattributedtoiowaitwhennoprocessesarereadyfortheCPUbutatleastoneprocessiswaitingonI/O.AhighpercentageofiowaittimeindicatesthatdiskI/Oisamajorcontributortothedelayinexecutionofprocesses.Ingeneral,ifsystemslownessoccursand%iowaitis20%to25%orhigher,investigationofadiskbottleneckisinorder.
%tm_actpercentageoftimethediskisbusy
NOTE:Hightm_actpercentagecanindicateadiskbottleneck.
When%tm_actortimeactiveforadiskishigh,noticeableperformance
degradationcanoccur.Onsomesystems,a%tm_actof35%orhigherforonediskcancausenoticeablyslowperformance.
oLookforbusyvs.idledrives.Movingdatafrommorebusytolessbusydrivesmayhelpalleviateadiskbottleneck.oCheckforpagingactivitybyfollowingtheinstructionsinthe"Memorybottlenecks"section.PagingtoandfromdiskwillcontributetotheI/Oload.
2.filemon
Tofindoutwhatfiles,logicalvolumes,anddisksaremostactive,runthefollowingcommandasroot:
#filemon-u-Oall-o/tmp/fmon.out;sleep30;trcstop
In30seconds,areportiscreatedin/tmp/fmon.out.
oCheckformostactivesegments,logicalvolumes,andphysicalvolumesinthisreport.oCheckforreadsandwritestopagingspacetodetermineifthediskactivityistrueapplicationI/Oorisduetopagingactivity.oCheckforfilesandlogicalvolumesthatareparticularlyactive.Iftheseareonabusyphysicalvolume,movingsomedatatoalessbusydiskmayimproveperformance.
TheMostActiveSegmentsreportliststhemostactivefilesbyfilesystemandinode.Themountpointofthefilesystemandinodeofthefilecanbeusedwiththencheckcommandtoidentifyunknownfiles:
#ncheck-i
Thisreportisusefulindeterminingiftheactivityistoafilesystem(segtype=persistent),theJFSlog(segtype=log),orto
-
8/2/2019 Tips for Performance Bottlenecks
6/25
pagingspace(segtype=working).
Byexaminingthereadsandreadsequencescounts,youcandetermineiftheaccessissequentialorrandom.Asthereadsequencescountapproachesthereadscount,fileaccessismorerandom.Thesameappliestothewritesandwritesequences.
------------------------------------------------------------------------
SMPperformancetuning
Performancetools
1.SMPonly(someSMPsdonotsupportthiscommand)
cpu_state-lThisdisplaysthecurrentstateofeachprocessor(enabled,disabled,orunavailable).
2.AIXtoolsthathavebeenadaptedinordertodisplaymoremeaninfulinformationonSMPsystemsps-m-oTHREADTheBNDcolumnwillindicatetheprocessornumbertowhichaprocess/threadisbound,ifitisbound.
pstat-ATheCPUIDcolumnwillindicatetheprocessornumbertowhichaprocess/threadisbound.
sar-PALLLoadonalltheprocessors.
vmstatDisplayskthr(kernelthreads).
netpmon-tPrintsCPUreportsonaper-threadbasis.
3.OtherAIXtoolsthatdidnotchange
filemoniostatsvmontprof
------------------------------------------------------------------------
Tuningmethodology
1.Checkavailablityofprocessors.
cpu_state-l
2.Checkbalancebetweenprocessors.
sar-PALL
3.Identifyboundprocesses/threads.
ps-m-oTHREAD
-
8/2/2019 Tips for Performance Bottlenecks
7/25
pstat-A
4.Unbindanyboundprocesses/threadsthatcanandshouldbeunbound.
5.Continueaswithuniprocessorsystem.
------------------------------------------------------------------------
Additionalinformation
1.KBUFFERSvs.VMM
TheBlockI/OBufferCache(KBUFFERS)isonlyusedwhendirectlyaccessingablockdevice,suchas/dev/hdisk0.NormalaccessthroughtheJournaledFileSystem(JFS)ismanagedbytheVirtualMemoryManager(VMM)anddoesnotusethetraditionalmethodforcachingthedatablocks.AnyI/OoperationstorawlogicalvolumesorphysicalvolumesdoesnotusetheBlockI/OBufferCache.
2.I/OPacing
UsersofAIXoccasionallyencounterlonginteractive-applicationresponsetimeswhenotherapplicationsinthesystemarerunninglargewritestodisk.Becausemostwritesareasynchronous,FIFOI/Oqueues
ofseveralmegabytesmaybuildupandtakeseveralsecondstocomplete.Theperformanceofaninteractiveprocessisseverelyimpactedifeverydiskreadspendsseveralsecondsworkingthroughthequeue.I/OpacinglimitsthenumberofoutstandingI/Orequestsagainstafile.Whenaprocesstriestowritetoafilewhosequeueisatthehigh-watermark(shouldbeamultipleof4plus1),theprocessissuspendeduntilenoughI/Oshavecompletedtobringthequeueforthatfiletothelow-watermark.Thedeltabetweenthehighandlowwatermarksshouldbekeptsmall.
ToconfigureI/OpacingonthesystemviaSMIT,enterthefollowingatthecommandlineasroot:
#smittychgsys
3.AsyncI/O
AsyncI/Oisperformedinthebackgroundanddoesnotblocktheuserprocess.ThisimprovesperformancebecauseI/Ooperationsandapplicationprocessingcanrunconcurrently.However,applicationsmustbespecificallywrittentotakeadvantageofAysncI/Othatismanagedbytheaiodaemonsrunningonthesystem.
ToconfigureAsyncI/OforthesystemviaSMIT,enterthefollowingatthecommandlineasroot:
#smittyaio
[TechDocsRef:90605198014824PublishDate:Dec.15,19994FAXRef:2445]PerformanceTuning--ThevmstatTool
------------------------------------------------------------------------
Contents
-
8/2/2019 Tips for Performance Bottlenecks
8/25
AboutthisdocumentRelateddocumentationAboutvmstatSummarystatistics------------------------------------------------------------------------
Aboutthisdocument
Thisdocumentprovidesanoverviewoftheoutputofthevmstat.ThisinformationappliestoAIXVersions4.x.
Relateddocumentation
Thefieldsproducedbythes,f,and[Drives]optionsofvmstatarefullydocumentedintheAIXPerformanceTuningGuide,publicationnumberSC23-2365,andintheonlineproductdocumentation.
TheAIXandRS/6000productdocumentationlibraryisalsoavailable:
www.rs6000.ibm.com/library/------------------------------------------------------------------------
Aboutvmstat
Althoughasystemmayhavesufficientrealresources,itmayperformbelowexpectationsiflogicalresourcesarenotallocatedproperly.
Usevmstattodeterminerealandlogicalresourceutilization.Itsampleskerneltablesandcountersandthennormalizestheresultsandpresentstheminanappropriateformat.
Bydefault,vmstatsendsitsreporttostandardout,butitcanberunwiththeoutputredirected.
vmstatisnormallyinvokedwithanintervalandacountspecified.Theintervalisthelengthoftimeinsecondsoverwhichvmstatistogatherandreportdata.Thecountisthenumberofintervalstorun.Ifno
parametersarespecified,vmstatwillreportasinglerecordofstatisticsforthetimesincethesystemwasbooted.Theremayhavebeeninactivityorfluctuationsintheworkload,sotheresultsmaynotrepresentcurrentactivity.Beawarethatthefirstrecordintheoutputpresentsstatisticssincethelastboot(exceptwheninvokedwiththe-for-soption).Inmanyinstances,thisdatacanbeignored.
vmstatreportsstatisticsaboutprocesses,virtualmemory,pagingactivity,faults,CPUactivity,anddisktransfers.Optionsandparametersrecognizedbythistoolareindicatedbytheusageprompt:
vmstat[-fs][Drives][Interval][Count]
Thefollowingfigurelistsoutputwherethesmallestworkunitiscalledakernelthread(kthr).Therandbunderthiscolumnrepresentthenumberof"threads",notprocesses,placedonthesequeues.
--------------------------------------------------------------------|||kthrmemorypagefaultscpu||-------------------------------------------------------||rbavmfrerepipofrsrcyinsycsussyidwa||0067471253000000114102201260|
-
8/2/2019 Tips for Performance Bottlenecks
9/25
|106747125300000011311843174790||0067471253000000118993383890|||--------------------------------------------------------------------Figure:Sampleoutputfromvmstat13
kthr
Thecolumnsunderthekthrheadingintheoutputprovideinformationabouttheaveragenumberofthreadsonvariousqueues.
r
Thercolumnindicatestheaveragenumberofkernelthreadsontherunqueueatone-secondintervals.
Thisfieldindicatedthenumberof"run-able"threads.Thesystemcountsthenumberofready-to-runthreadsoncepersecondandaddsthatnumbertoaninternalcounter.vmstatthensubtractstheinitialvalueofthiscounterfromtheendvalueanddividestheresultbythenumberofsecondsinthemeasurementinterval.Thisvalueistypicallylessthanfivewithastableworkload.Ifthisvalueincreasesrapidly,lookforanapplicationproblem.Iftherearemanythreads(especiallyCPU-intensiveones)competingfortheCPUresource,itis
quitepossibletheywillbescheduledinround-robinfashion.Ifeachoneexecutesforacompleteorpartialtimeslice,thenumberof"run-able"threadscouldeasilyexceed100.
b
Thebcolumnshowstheaveragenumberofkernelthreadsonthewaitqueueatone-secondintervals(awaitingresource,awaitinginput/output).
Kernelthreadsareplacedonthewaitqueuewhenscheduledforexecutionandarewaitingforoneoftheirprocesspagestobepagedin.Onceasecond,thesystemcountsthethreadswaitingandaddsthat
numbertoaninternalcounter.vmstatthensubtractstheinitialvaluefromtheendvalueanddividestheresultbythenumberofsecondsinthemeasurementinterval.Thisvalueisusuallynearzero.Donotconfusethiswithwa--waitingoninput/output(I/O).
NOTE:OnanSMPsystem,therewillbeanadditionalblockedprocessshowninthebcolumn.ThisisforthelrudkprocthatispartoftheVirtualMemoryManager's(VMM)page-replacementalgorithm.
Also,onasystemwithacompressedjournaledfilesystem(JFS)thatismounted,therewillbeanadditionalblockedprocess:thejfsckproc.
memory
Theinformationunderthememoryheadingprovidesinformationaboutrealandvirtualmemory.
avm
Theavmcolumngivestheaveragenumberofpagesallocatedtopagingspace.(InAIX,apagecontains4096bytesofdata.)
-
8/2/2019 Tips for Performance Bottlenecks
10/25
Whenaprocessexecutes,spaceforworkingstorageisallocatedonthepagingdevices(backingstore).Thiscanbeusedtocalculatetheamountofpagingspaceassignedtoexecutingprocesses.Thenumberintheavmfielddividedby256willyieldthenumberofmegabytes(MB),systemwide,allocatedtopagespace.Thelsps-acommandalsoprovidesinformationonindividualpagingspace.Itisrecommendedthatenoughpagingspacebeconfiguredonthesystemsothatthepagingspaceuseddoesnotapproach100percent.Whenfewerthan128unallocatedpagesremainonthepagingdevices,thesystemwillbegintokillprocessestofreesomepagingspace.
VersionsofAIXbefore4.3.2allocatedpagingspaceblocksforpagesofmemoryasthepageswereaccessed.Onalargememorymachine,wheretheapplicationsetissuchthatpagingisneverorrarelyrequired,thesepagingspaceblockswereallocatedbutneverneeded.AIXVersion4.3.2implementsdeferredpagingspaceallocation,inwhichthepagingspaceblocksarenotallocateduntilpagingisnecessary,thus,helpingreducethepagingspacerequirementsofthesystem.Theavmvalueinvmstatindicatesthenumberofvirtualmemory(workingstorage)pagesthathavebeenaccessedbutnotnecessarilypagedout.Withthepreviouspolicyof"latepagespaceallocation",avmhadthesamedefinition.However,sincetheVMMwouldallocatepagingspacediskblocksforeachworkingpagethatwasaccessed,thepagingspaceblockswasequaltotheavm.Thereasonforthepagingspaceblocksto
beallocatedatthetimetheworkingpagesareaccessedissothatifthepageshadtobepagedoutofmemory,therewouldbediskblocksonthepagespacelv'savailableforthein-memorypagestogo.Onsystemsthatneverpage-outtopage-space,it'sawasteofdiskspacetohaveasmanypagespacediskblocksasthereismemory.Withdeferredpolicy,thepagespacediskblocksareonlyallocatedforthepagesthatdoneedtobepagedout.Theavmnumberwillgrowasmoreprocessesgetstartedand/orexistingprocessesusemoreworkingstorage.Likewise,thenumberwillshrinkasprocessesexitand/orfreeworkingstorage.
fre
Thefrecolumnshowstheaveragenumberoffreememoryframes.Aframeisa4096-byteareaofrealmemory.
Thesystemmaintainsabufferofmemoryframes,calledthefreelist,thatwillbereadilyaccessiblewhentheVMMneedsspace.Thenominalsizeofthefreelistvariesdependingontheamountofrealmemoryinstalled.Onsystemswith64MBofmemoryormore,theminimumvalue(MINFREE)is120frames.Forsystemswithlessthan64MB,thevalueistwotimesthenumberofMBofrealmemory,minus8.Forexample,asystemwith32MBwouldhaveaMINFREEvalueof56freeframes.
IfthefrevalueissubstantiallyabovetheMAXFREEvalue(whichisdefinedasMINFREEplus8),thenitisunlikelythatthesystemis
thrashing(continuouslypaginginandout).However,ifthesystemisthrashing,beassuredthatthefrevalueissmall.MostUNIXandAIXoperatingsystemswillusenearlyallavailablememoryfordiskcaching,soyouneednotbealarmedifthefrevalueoscillatesbetweenMINFREEandMAXFREE.
page
Theinformationunderthepageheadingincludesinformationaboutpagefaultsandpagingactivity.
-
8/2/2019 Tips for Performance Bottlenecks
11/25
re
Therecolumnshowsthenumber(rate)ofpagesreclaimed.
ReclaimedpagescansatisfyanaddresstranslationfaultwithoutinitiatinganewI/Orequest(thepageisstillinmemory).Thisincludespagesthathavebeenputonthefreelistbutareaccessedagainbeforetheyarereassigned.ItincludespagespreviouslyrequestedbyVMMforwhichI/Ohasnotyetbeencompletedorthosepre-fetchedbyVMM'sread-aheadmechanismbuthiddenfromthefaultingsegment.
pi
Thepicolumndetailsthenumber(rate)ofpagespagedinfrompagingspace.
Pagingspaceisthepartofvirtualmemorythatresidesondisk.Itisusedasanoverflowwhenmemoryisovercommitted.Pagingconsistsofpaginglogicalvolumesdedicatedtothestorageofworkingsetpagesthathavebeenstolenfromrealmemory.Whenastolenpageisreferencedbytheprocess,apagefaultoccursandthepagemustbereadintomemoryfrompagingspace.Thereisno"good"numberforthis
duetothevarietyofconfigurationsofhardware,software,andapplications.
Onetheoryisthatfivepage-inspersecondshouldbetheupperlimit.Usethistheoreticalmaximumasareferencebutdonotadheretoitrigidly.Thisfieldisimportantasakeyindicatorofpagingspaceactivity.Ifapage-inoccurs,thentheremusthavebeenapreviouspage-outforthatpage.Itisalsolikelyinamemory-constrainedenvironmentthateachpage-inwillforceadifferentpagetobestolenand,therefore,pagedout.
po
Thepocolumnshowsthenumber(rate)ofpagespagedouttopagingspace.
Wheneverapageofworkingstorageisstolen,itiswrittentopagingspace.Ifnotreferencedagain,itremainsonthepagingdeviceuntiltheprocessterminatesordisclaimsthespace.Subsequentreferencestoaddressescontainedwithinthefaulted-outpagesresultinpagefaults,andthepagesarepagedinindividuallybythesystem.Whenaprocessterminatesnormally,anypagingspaceallocatedtothatprocessisfreed.Ifthesystemisreadinginasignificantnumberofpersistentpages,youmayseeanincreaseinpowithoutcorrespondingincreasesinpi.Thissituationdoesnotnecessarilyindicatethrashing,butmaywarrantinvestigationintodataaccesspatternsof
theapplications.
fr
Thefrcolumndetailsthenumber(rate)ofpagesfreed.
AstheVMMpage-replacementcoderoutinescansthePageFrameTable(PFT),itusescriteriatoselectwhichpagesaretobestolentoreplenishthefreelistofavailablememoryframes.ThetotalpagesstolenbytheVMM--bothworking(computational)andfile
-
8/2/2019 Tips for Performance Bottlenecks
12/25
(persistent)pages--arereportedasaratepersecond.JustbecauseapagehasbeenfreeddoesnotmeanthatanyI/Ohastakenplace.Forexample,ifapersistentstorage(file)pagehasnotbeenmodified,itwillnotbewrittenbacktothedisk.IfI/Oisnotnecessary,minimalsystemresourcesarerequiredtofreeapage.
sr
Thesrcolumndetailsthenumber(rate)ofpagesscannedbythepage-placementalgorithm.
TheVMMpage-replacementcodescansthePFTandstealspagesuntilthenumberofframesonthefreelistisatleasttheMAXFREEvalue.Thepage-replacementcodemayhavetoscanmanyentriesinthePageFrameTablebeforeitcanstealenoughtosatisfythefreelistrequirements.Withstable,unfragmentedmemory,thescanrateandfreeratemaybenearlyequal.Onsystemswithmultipleprocessesusingmanydifferentpages,thepagesaremorevolatileanddisjointed.Inthisscenario,thescanratemaygreatlyexceedthefreerate.
cy
ThecycolumnprovidestherateofcompletescansofthePageFrameTable.
cyshowshowmanytimes(persecond)thepage-replacementcodehasscannedthePageFrameTable.SincethefreelistcanbereplenishedwithoutacompletescanofthePFTandbecauseallofthevmstatfieldsarereportedasintegers,thisfieldisusuallyzero.
faults
Theinformationunderthefaultsheadinginthevmstatoutputprovidesinformationaboutprocesscontrol.
in
Theincolumnshowsthenumber(rate)ofdeviceinterrupts.
Thiscolumnshowsthenumberofhardwareordeviceinterrupts(persecond)observedoverthemeasurementinterval.Examplesofinterruptsarediskrequestcompletionsandthe10millisecondclockinterrupt.Sincethelatteroccurs100timespersecond,theinfieldisalwaysgreaterthan100.
sy
Thesycolumndetailsthenumber(rate)ofsystemcalls.
Resourcesareavailabletouserprocessesthroughwell-definedsystem
calls.Thesecallsinstructthekerneltoperformoperationsforthecallingprocessandexchangedatabetweenthekernelandtheprocess.Sinceworkloadsandapplicationsvaryanddifferentcallsperformdifferentfunctions,itisimpossibletosayhowmanysystemcallspersecondaretoomany.
cs
Thecscolumnshowsthenumber(rate)ofcontextswitches.
-
8/2/2019 Tips for Performance Bottlenecks
13/25
ThephysicalCPUresourceissubdividedintologicaltimeslicesof10millisecondseach.Assumingaprocessisscheduledforexecution,itwillrununtilitstimesliceexpires,itispreempted,oritvoluntarilygivesupcontroloftheCPU.WhenanotherprocessisgivencontroloftheCPU,thecontext,orworkingenvironment,ofthepreviousprocessmustbesavedandthecontextofthecurrentprocessmustbeloaded.AIXhasaveryefficientcontextswitchingprocedure,soeachswitchisinexpensiveintermsofresources.Anysignificantincreaseincontextswitchesiscauseforfurtherinvestigation.
cpu
TheinformationunderthecpuheadinginthevmstatoutputprovidesabreakdownofCPUusage.
us
TheuscolumnshowsthepercentofCPUtimespentinusermode.
Processesexecuteinusermodeorsystem(kernel)mode.Wheninusermode,aprocessexecuteswithinitscodeanddoesnotrequirekernelresourcestoperformcomputations,managememory,setvariables,andsoon.
sy
ThesycolumndetailsthepercentofCPUtimespentinsystemmode.
Ifaprocessneedskernelresources,itmustexecuteacallandgointosystemmodetomakethatresourceavailable.I/Otoadrive,forexample,requiresacalltoopenthedevice,seek,andreadandwritedata.ThisfieldshowsthepercentoftimetheCPUwasinsystemmode.OptimumusewouldhavetheCPUworking100percentofthetime.Thisholdstrueinthecaseofasingle-usersystemwithnoneedtosharetheCPU.Generally,ifus+sytimeisbelow90percent,asingle-usersystemisnotconsideredCPUconstrained.However,ifus+sytimeonamulti-usersystemexceeds80percent,theprocessesmayspendtime
waitingintherunqueue.Responsetimeandthroughputmightsuffer.
id
Iftherearenoprocessesavailableforexecution(therunqueueisempty),thesystemdispatchesaprocesscalledwait.Thepsreport(withthe-korgoption)identifiesthisaskprocwithaprocessID(PID)of516.Donotworryifyourpsreportshowsahighaggregatetimeforthisprocess.Itmeansyouhavehadsignificantperiodsoftimewhennootherprocessescouldrun.IftherearenoI/Ospendingtoalocaldisk,alltimechargedtothewaitprocessisclassifiedasidletime.
wa
ThewacolumndetailsCPUidletime(percent)withpendinglocaldiskI/O.
IfthereisatleastoneoutstandingI/Otoalocaldiskwhenthewaitprocessisrunning,thetimeisclassifiedas"waitingonI/O".Awavalueover40percentcouldindicatethatthedisksubsystemmaynotbebalancedproperly,oritmaybetheresultofadisk-intensiveworkload.Ifthereisonlyoneprocessavailableforexecution--
-
8/2/2019 Tips for Performance Bottlenecks
14/25
oftenthecaseonatechnicalworkstation--theremaybenowaytoavoidwaitingonI/O.
NOTE:ThewacolumnonSMPmachinesrunningAIXVersion4.3.2orearlierissomewhatexaggerated.Thisisduetothemethodusedincalculatingwio.
MethodusedinAIX4.3.2andearlierAIXVersions
Ateachclockinterruptoneachprocessor(100timesasecondinAIX),adeterminationismadeastowhichoffourcategories(usr/sys/wio/idle)toplacethelast10millisecondsoftime.IftheCPUwasbusyinusrmodeatthetimeoftheclockinterrupt,thenusrgetstheclocktickaddedintoitscategory.IftheCPUwasbusyinkernelmodeatthetimeoftheclockinterrupt,thenthesyscategorygetsthetick.IftheCPUwasNOTbusy,thenacheckismadetoseeifanyI/Otodiskisinprogress.IfanydiskI/Oisinprogress,thenthewiocategoryisincremented.IfNOdiskI/OisinprogressandtheCPUisnotbusy,thentheidlcategorygetsthetick.
------------------------------------------------------------------------
Summarystatistics
vmstatwiththe-soptionreportsabsolutecountsofvariouseventssincethesystemwasbooted.Thereare23separateeventsreportedinthevmstat-soutput;thefollowing4haveprovenmosthelpful.The19remainingfieldscontainavarietyofactivitiesfromaddresstranslationfaultstolockmissestosystemcalls.Theinformationinthose19fieldsisalsovaluablebutislessfrequentlyused.
pageins
Thepageinsfieldshowsthenumbersystemwidepage-ins.
Whenapageisreadfromdisktomemory,thiscountisincremented.ItisacountofVMM-initiatedreadoperationsand,withthepageoutsfield,
representstherealI/O(diskreadsandwrites)initiatedbytheVMM.
pageouts
Thepageoutsfieldshowsthenumberofsystemwidepage-outs.
Theprocessofwritingpagestothediskiscountincremented.ThepageoutsfieldvalueisatotalcountofVMM-initiatedwriteoperationsand,withthepageinsfield,representsthetotalamountofrealI/OinitiatedbytheVMM.
pagingspacepageins
ThepagingspacepageinsfieldisthecountofONLYpagesreadfrompagingspace.
pagingspacepageouts
ThepagingspacepageoutsfieldisthecountofONLYpageswrittentopagingspace.
Usingthesummarystatistics
-
8/2/2019 Tips for Performance Bottlenecks
15/25
Thefourprecedingfieldscanbeusedtoindicatehowmuchofthesystem'sI/Oisforpersistentstorage.Ifthevalueforpagingspacepageinsissubtractedfromthe(systemwide)valueforpageins,theresultisthenumberofpagesthatwerereadfrompersistentstorage(files).Likewise,ifthevalueforpagingspacepageoutsissubtractedfromthe(systemwide)valueforpageouts,theresultisthenumberofpersistentpages(files)thatwerewrittentodisk.
Rememberthatthesecountsapplytothetimesincesysteminitialization.Ifyouneedcountsforagiventimeinterval,executevmstat-satthetimeyouwanttostartmonitoringandagainattheendoftheinterval.Thedeltasbetweenlikefieldsofsuccessivereportswillbethecountfortheinterval.Itiseasiertoredirecttheoutputofthereportstoafileandthenperformthemath.
[TechDocsRef:90605226914708PublishDate:Nov.04,19994FAXRef:6220]
PerformanceTuning--TheiostatTool
Contents
AboutthisdocumentSyntax
TTYstatisticsintheiostatoutputCPUstatisticsintheiostatoutputDiskstatisticsintheiostatoutputAnalyzingthedataConclusions
Aboutthisdocument
Thisdocumentisbasedon"PerformanceTuning:AContinuingSeries--TheiostatTool",byBarrySaad,fromtheJanuary/February1994issueofAIXTRA:IBM'SMAGAZINEFORAIXPROFESSIONALS.
ThisarticlediscussesiostatandhowitcanidentifyI/O-subsystemandCPU
bottlenecks.iostatworksbysamplingthekernel'saddressspaceandextractingdatafromvariouscountersthatareupdatedeveryclocktick(1clocktick=10milliseconds[ms]).Theresults--coveringTTY,CPU,andI/Osubsystemactivity--arereportedasper-secondratesorasabsolutevaluesforthespecifiedinterval.ThisdocumentappliestoAIXVersions4.x.
------------------------------------------------------------------------
Syntax
Normally,iostatisissuedwithbothanintervalandacountspecified,withthereportsenttostandardoutputorredirectedtoafile.The
commandsyntaxappearsbelow:
iostat[-t][-d][Drives][Interval[Count]]
The-dflagcausesiostattoprovideonlydiskstatisticsforalldrives.The-tflagcausesiostattoprovideonlysystem-wideTTYandCPUstatistics.
NOTE:The-tand-doptionsaremutuallyexclusive.
-
8/2/2019 Tips for Performance Bottlenecks
16/25
Ifyouspecifyoneormoredrives,theoutputislimitedtothosedrives.Multipledrivescanbespecified;separatetheminthesyntaxwithspaces.
Youcanspecifyatimeinsecondsfortheintervalbetweenrecordstobeincludedinthereports.Theinitialrecordcontainsstatisticsforthetimesincesystemboot.Succeedingrecordscontaindatafortheprecedinginterval.Ifnointervalisspecified,asinglerecordisgenerated.
Ifyouspecifyaninterval,thecountofthenumberofrecordstobeincludedinthereportcanalsobespecified.Ifyouspecifyanintervalwithoutacount,iostatwillcontinuerunninguntilitiskilled.
--------------------------------------------------------------------|||tty:tintoutavg-cpu:%user%sys%idle%iowait||2.23.30.41.397.70.6||||Disks:%tm_actKbpstpsmspsKb_readKb_wrtn||hdisk00.41.10.31176751087266||hdisk10.31.00.2592301017734||hdisk20.00.20.018018946832||cd00.00.00.000||||tty:tintoutavg-cpu:%user%sys%idle%iowait|
|2.23.30.41.397.70.6||||Disks:%tm_actKbpstpsmspsKb_readKb_wrtn||hdisk00.41.10.31176751087266||hdisk10.31.00.2592301017734||hdisk20.00.20.018018946832||cd00.00.00.000|||--------------------------------------------------------------------Figure1.SampleOutputfromiostat22
Thefollowingsectionsexplaintheoutput.------------------------------------------------------------------------
TTYstatisticsintheiostatoutput
ThetwocolumnsofTTYinformation(tinandtout)intheiostatoutputshowthenumberofcharactersreadandwrittenbyallTTYdevices,includingbothrealandpseudoTTYdevices.RealTTYdevicesarethoseconnectedtoanasynchronousport.Some"pseudoTTYdevices"areshells,telnetsessions,andaixterms.
*tinshowsthetotalcharacterspersecondreadbyallTTYdevices.*toutindicatesthetotalcharacterspersecondwrittentoallTTYdevices.
Generallytherearefewerinputcharactersthanoutputcharacters.Forexample,assumeyourunthefollowing:
iostat-t130cd/usr/sbinls-l
Youwillseefewinputcharactersandmanyoutputcharacters.Ontheotherhand,applicationssuchasviresultinasmallerdifferencebetweenthenumberofinputandoutputcharacters.Analystsusingmodemsfor
-
8/2/2019 Tips for Performance Bottlenecks
17/25
asynchronousfiletransfermaynoticethenumberofinputcharactersexceedingthenumberofoutputcharacters.Naturally,thisdependsonwhetherthefilesarebeingsentorreceivedrelativetothemeasuredsystem.
SincetheprocessingofinputandoutputcharactersconsumesCPUresource,lookforacorrelationbetweenincreasedTTYactivityandCPUutilization.Ifsucharelationshipexists,evaluatewaystoimprovetheperformanceoftheTTYsubsystem.Stepsthatcouldbetakenincludechangingtheapplicationprogram,modifyingTTYportparametersduringfiletransfer,orperhapsupgradingtoafasterormoreefficientasynchronouscommunicationsadapter.
------------------------------------------------------------------------
CPUstatisticsintheiostatoutput
TheCPUstatisticscolumns(%user,%sys,%idle,and%iowait)provideabreakdownofCPUusage.Thisinformationisalsoreportedintheoutputofthevmstatcommandinthecolumnslabeledus,sy,id,andwa.
%user
The%usercolumnshowsthepercentageofCPUresourcespentinusermode.
AUNIXprocesscanexecuteinuserorsystemmode.Wheninusermode,aprocessexecuteswithinitsowncodeanddoesnotrequirekernelresources.
%sys
The%syscolumnshowsthepercentageofCPUresourcespentinsystemmode.ThisincludesCPUresourceconsumedbykernelprocesses(kprocs)andothersthatneedaccesstokernelresources.Forexample,thereadingorwritingofafilerequireskernelresourcestoopenthefile,seekaspecificlocation,andreadorwritedata.AUNIXprocessaccesseskernelresourcesbyissuingsystemcalls.
Typically,theCPUispacing(thesystemisCPUbound)ifthesumofuser
andsystemtimeexceeds90percentofCPUresourceonasingle-usersystemor80percentonamulti-usersystem.ThisconditionmeansthattheCPUisthelimitingfactorinsystemperformance.
Theratioofusertosystemmodeisdeterminedbyworkloadandismoreimportantwhentuninganapplicationthanwhenevaluatingperformance.
AkeyfactorwhenevaluatingCPUperformanceisthesizeoftherunqueue(providedbythevmstatcommand).Ingeneral,astherunqueueincreases,userswillnoticedegradation(anincrease)inresponsetime.
%idle
The%idlecolumnshowsthepercentageofCPUtimespentidle,orwaiting,withoutpendinglocaldiskI/O.Iftherearenoprocessesontherunqueue,thesystemdispatchesaspecialkernelprocesscalledwait.OnmostAIXsystems,thewaitprocessID(PID)is516.
%iowait
The%iowaitcolumnshowsthepercentageoftimetheCPUwasidlewithpendinglocaldiskI/O.
-
8/2/2019 Tips for Performance Bottlenecks
18/25
TheiowaitstateisdifferentfromtheidlestateinthatatleastoneprocessiswaitingforlocaldiskI/Orequeststocomplete.UnlesstheprocessisusingasynchronousI/O,anI/Orequesttodiskcausesthecallingprocesstoblock(orsleep)untiltherequestiscompleted.Onceaprocess'sI/Orequestcompletes,itisplacedontherunqueue.
Ingeneral,ahighiowaitpercentageindicatesthesystemhasamemoryshortageoraninefficientI/Osubsystemconfiguration.UnderstandingtheI/ObottleneckandimprovingtheefficiencyoftheI/Osubsystemrequiremoredatathaniostatcanprovide.However,typicalsolutionsmightinclude:
*limitingthenumberofactivelogicalvolumesandfilesystemsplacedonaparticularphysicaldisk(TheideaistobalancefileI/Oevenlyacrossallphysicaldiskdrives.)*spreadingalogicalvolumeacrossmultiplephysicaldisks(Thisisusefulwhenanumberofdifferentfilesarebeingaccessed.)*creatingmultipleJFSlogsforavolumegroupandassigningthemtospecificfilesystems(Thisisbeneficialforapplicationsthatcreate,delete,ormodifyalargenumberoffiles,particularlytemporaryfiles.)*backingupandrestoringfilesystemstoreducefragmentation(Fragmentationcausesthedrivetoseekexcessivelyandcanbealargeportionofoverallresponsetime.)
*addingadditionaldrivesandrebalancingtheexistingI/Osubsystem
Onsystemsrunningaprimaryapplication,highI/Owaitpercentagemayberelatedtoworkload.Inthiscase,theremaybenowaytoovercometheproblem.Onsystemswithmanyprocesses,somewillberunningwhileotherswaitforI/O.Inthiscase,theiowaitcanbesmallorzerobecauserunningprocesses"hide"waittime.Althoughiowaitislow,abottleneckmaystilllimitapplicationperformance.TounderstandtheI/Osubsystemthoroughly,examinethestatisticsinthenextsection.
NOTE:The%iowaitcolumnonSMPmachinesrunningAIXVersions4.3.2orearlierissomewhatexaggerated.Thisisduetothemethodusedincalculatingwio.
MethodusedinAIX4.3.2andearlierAIXVersions
Ateachclockinterruptoneachprocessor(100timesasecondinAIX),adeterminationismadeastowhichoffourcategories(usr/sys/wio/idle)toplacethelast10msoftime.IftheCPUwasbusyinusrmodeatthetimeoftheclockinterrupt,thenusrgetstheclocktickaddedintoitscategory.IftheCPUwasbusyinkernelmodeatthetimeoftheclockinterrupt,thenthesyscategorygetsthetick.IftheCPUwasNOTbusy,thenacheckismadetoseeifanyI/Otodiskisinprogress.IfanydiskI/Oisinprogress,thenthewiocategoryisincremented.IfNOdiskI/OisinprogressandtheCPUisnotbusy,thentheidlcategorygetsthetick.------------------------------------------------------------------------
Diskstatisticsintheiostatoutput
ThediskstatisticsportionoftheiostatoutputprovidesabreakdownofI/Ousage.Thisinformationisusefulindeterminingwhetheraphysicaldiskislimitingperformance.
DiskI/Ohistory
Thesystemmaintainsahistoryofdiskactivitybydefault.Notethat
-
8/2/2019 Tips for Performance Bottlenecks
19/25
historyisdisabledifyouseethemessage:
Diskhistorysincebootnotavailable.
Thismessagedisplaysonlyinthefirstoutputrecordfromiostat.
DiskI/OhistoryshouldbeenabledsincetheCPUresourceusedinmaintainingitisinsignificant.History-keepingcanbedisabledorenabledinSMITunderthefollowingpath:
->SystemEnvironments->Change/ShowCharacteristicsofOperatingSystem->ContinuouslymaintainDISKI/Ohistory->true|false
Choosetruetoenablehistory-keepingorfalsetodisableit.
Disks
TheDisks:columnshowsthenamesofthephysicalvolumes.Theyareeitherhdiskorcdfollowedbyanumber.(hdisk0andcd0refertothefirstphysicaldiskdriveandthefirstCDdiskdrive,respectively.)
%tm_act
The%tm_actcolumnshowsthepercentageoftimethevolumewasactive.Thisistheprimaryindicatorofabottleneck.
Adriveisactiveduringdatatransferandcommandprocessing,suchasseekingtoanewlocation.Thedisk-usepercentageisdirectlyproportionaltoresourcecontentionandinverselyproportionaltoperformance.Asdiskuseincreases,performancedecreasesandresponsetimeincreases.Ingeneral,whenadisk'suseexceeds70percent,processesarewaitinglongerthannecessaryforI/OtocompletebecausemostUNIXprocessesblock(orsleep)whilewaitingfortheirI/Orequeststocomplete.
Kbps
KbpsshowstheamountofdatareadfromandwrittentothedriveinKBspersecond.ThisisthesumofKb_readplusKb_wrtn,dividedbythenumberofsecondsinthereportinginterval.
tps
tpsreportsthenumberoftransferspersecond.AtransferisanI/Orequestatthedevicedriverlevel.
Kb_read
Kb_readreportsthetotaldata(inKBs)readfromthephysicalvolume
duringthemeasuredinterval.
Kb_wrtn
Kb_wrtnshowstheamountofdata(inKBs)writtentothephysicalvolumeduringthemeasuredinterval.
------------------------------------------------------------------------
Analyzingthedata
-
8/2/2019 Tips for Performance Bottlenecks
20/25
Takenalone,thereisnounacceptablevalueforanyoftheprecedingfieldsbecausestatisticsaretoocloselyrelatedtoapplicationcharacteristics,systemconfiguration,andtypesofphysicaldiskdrivesandadapters.Therefore,whenevaluatingdata,youmustlookforpatternsandrelationships.Themostcommonrelationshipisbetweendiskutilizationanddatatransferrate.
Todrawanyvalidconclusionsfromthisdata,youmustunderstandtheapplication'sdiskdataaccesspatterns--sequential,random,oracombination--andthetypeofphysicaldiskdrivesandadaptersonthesystem.
Forexample,ifanapplicationreadsandwritessequentially,youshouldexpectahighdisk-transferratewhenyouhaveahighdisk-busyrate.(NOTE:Kb_readandKb_wrtncanconfirmanunderstandingofanapplication'sreadandwritebehaviorbuttheyprovidenoinformationonthedataaccesspatterns.)
Generallyyoudonotneedtobeconcernedaboutahighdisk-busyrateaslongasthedisk-transferrateisalsohigh.However,ifyougetahighdisk-busyrateandalowdata-transferrate,youmayhaveafragmentedlogicalvolume,filesystem,orindividualfile.
Whatisahighdata-transferrate?Thatdependsonthediskdriveandtheeffectivedata-transferrateforthatdrive.Youshouldexpectnumbersbetweentheeffectivesequentialandeffectiverandomdisk-transferrates.BelowisachartofeffectivetransferratesforseveralcommonSCSI-1andSCSI-2diskdrives.
Table1.EffectiveTransferRate(KB/sec),Part1of2TYPEOFACCESS400MBDRIVE670MBDRIVE857MBDRIVERead-Sequential158915252142Read-Random241172262Write-Sequential118511081588Write-Random327275367Table2.EffectiveTransferRate(KB/sec),Part2of2
TYPEOFACCESS1.2GBDRIVE1.37GBDRIVE1.2GBDRIVE1.37GBS-2DRIVERead-Sequential2169266721803123Read-Random292299385288Write-Sequential1464218921562357Write-Random362491405549
Thetransferratesweredeterminedduringperformancetestingandgivemoreaccurateexpectationsofdiskperformancethanthemedia-transferrate,whichreflectsthehardwarecapabilityanddoesnotaccountforoperatingsystemandapplicationoverhead.
Anotheruseofthedataistoanswerthequestion:"DoIneedanotherSCSIadapter?"Ifyou'veeverbeenaskedthisquestion,youprobablyprovideda
genericanswerorjustplainguessed.
Youcanusedatacapturedbyiostattoanswerthequestionaccuratelybytrackingtransferrates,findingthemaximumdata-transferrateforeachdisk.Assumethatthemaximumrateoccurssimultaneouslyforalldrives(theworstcase).Formaximumaggregateperformance,themeasuredtransferratesfordrivesattachedtoagivenadaptermustbebelowtheeffectiveSCSIadapterthroughputrating.
Forplanningpurposes,youshoulduse70percentoftheadapter'srated
-
8/2/2019 Tips for Performance Bottlenecks
21/25
throughput(forexample,2.8MBpersecondforaSCSI-1adapter).Thispercentageshouldprovideasufficientbufferforoccasionalpeakratesthatmayoccur.Whenaddingadrive,youmustassumethedata-transferrate.Atleastyouwillhavethecollecteddataandtheeffectivetransferratestouseasabasis.
KeepinmindthattheSCSIadaptermaybesaturatedifthedata-transferratesovermultipleintervalsapproachtheeffectiveSCSIadapterthroughputrating.Inthatcase,theprecedinganalysisisinvalid.
------------------------------------------------------------------------
Conclusions
TheprimarypurposeoftheiostattoolistodetectI/Obottlenecksbymonitoringthediskutilization(%tm_actfield).iostatcanalsobeusedtoidentifyCPUproblems,assistincapacityplanning,andprovideinsightintosolvingI/Oproblems.Armedwithbothvmstatandiostat,youcancapturethedatarequiredtoidentifyperformanceproblemsrelatedtoCPU,memory,andI/Osubsystems.
[TechDocsRef:90605205314704PublishDate:Oct.15,19994FAXRef:9779]
TheAIXVirtualMemoryManager(VMM)
------------------------------------------------------------------------
Contents
AboutthisdocumentVMMoverviewReal-MemorymanagementFreelistPersistentvs.memorysegmentsPagingspaceandvirtualmemoryVMMmemoryloadcontrolfacility
VMSTAT'savmfieldVMSTAT'sfrefieldHowthesystemisusingmemoryExplanationofsvmonoutput------------------------------------------------------------------------
Aboutthisdocument
ThisdocumentaddresseshowRAMandpagingspaceareused.ThisinformationappliestoAIXVersion4.x.------------------------------------------------------------------------
VMMoverview
TheVirtualMemoryManager(VMM)servicesmemoryrequestsfromthesystemanditsapplications.Virtual-memorysegmentsarepartitionedinunitscalledpages;eachpageiseitherlocatedinphysicalmemory(RAM)orstoredondiskuntilitisneeded.AIXusesvirtualmemoryinordertoaddressmorememorythanisphysicallyavailableinthesystem.ThemanagementofmemorypagesinRAMorondiskishandledbytheVMM.------------------------------------------------------------------------
Real-Memorymanagement
-
8/2/2019 Tips for Performance Bottlenecks
22/25
InAIX,virtual-memorysegmentsarepartitionedinto4096-byteunitscalledpages.Realmemoryisdividedinto4096-bytepageframes.TheVMMhastwomajorfunctions:1)managetheallocationofpageframes,and2)resolvereferencestovirtual-memorypagesthatarenotcurrentlyinRAM(storedinpagingspace)ordonotyetexist.
Inordertoaccomplishitstask,theVMMmaintainsafreelistofavailablepageframes.TheVMMalsousesapage-replacementalgorithmtodeterminewhichvirtual-memorypagescurrentlyinRAMwillhavetheirpageframesreassignedtothefreelist.Thepage-replacementalgorithmtakesintoaccounttheexistenceofpersistentvs.workingsegments,repaging,andVMMthresholds.------------------------------------------------------------------------
Freelist
TheVMMmaintainsalistoffreepageframesthatitusestosatisfypagefaults.Thefreelistismadeupofunallocatedpageframes.AIXtriestouseallofRAMallofthetime,exceptforasmallamountwhichitmaintainsonthefreelist.TomaintainthissmallamountofunallocatedpagestheVMMwillusePageOutsandPageStealstofreeupspaceandreassignthosepageframestothefreelist.Thevirtual-memorypageswhosepageframesaretobereassignedareselectedviatheVMM's
page-replacementalgorithm.------------------------------------------------------------------------
Persistentvs.workingmemorysegments
AIXdistinguishesbetweendifferenttypesofmemorysegments,andtounderstandtheVirtualMemoryManager,itisimportanttounderstandthedifferencebetweenworkingandpersistentsegments.Apersistentsegmenthasapermanentstoragelocationondisk.Filescontainingdataorexecutableprogramsaremappedtopersistentsegments.WhenaJournaledFileSystem(JFS)fileisopenedandaccessedthefiledataiscopiedintoRAM.VMMparameterscontrolwhenphysicalmemoryframesallocatedtopersistentpagesshouldbeoverwrittenandusedtostoreotherdata.
Workingsegmentsaretransitoryandexistonlyduringtheirusebyaprocess,andhavenopermanentdiskstoragelocation.Processstackanddataregionsaremappedtoworkingsegmentsandsharedlibrarytextsegments.Pagesofworkingsegmentsmustalsohavediskstoragelocationstooccupywhentheycannotbekeptinrealmemory.Thediskpagingspaceisusedforthispurpose.Whenaprogramexits,allofitsworkingpagesareplacedbackonthefreelistimmediately.------------------------------------------------------------------------
Pagingspaceandvirtualmemory
WorkingpagesinRAMthatcanbemodifiedandpagedoutareassigneda
correspondingslotinpagingspace.Theallocatedpagingspacewillonlybeusedifthepageneedstobepagedout.However,anallocatedpageinpagingspacecannotbeusedbyanotherpage.Itremainsreservedforaparticularpageforaslongasitexistsinvirtual-memory.Sincepersistentpagesarepagedouttotheirlocationondiskfromwhichtheycame,pagingspacedoesnotneedtobeallocatedforpersistentpagesresidinginRAM.
TheVMMhasthreemodesforallocatingpagingspace:earlypagingspaceallocation,latepagingspaceallocationanddeferredpagingspace
-
8/2/2019 Tips for Performance Bottlenecks
23/25
allocation.Earlyallocationpolicyreservespagingspacewheneveramemoryrequestforaworkingpageismade.Lateallocationpolicyonlyassignspagingspacewhentheworkingpageisreferenced.AIXVersions4.3.2andlaterimplementadeferredpagingspaceallocationpolicy.Withdeferredpagingspacepolicy,pagingspaceblocksarenotallocateduntiltheworkingpagesareactuallypagedoutofmemory.Thissignificantlyreducesthepagingspacerequirementsofthesystem.------------------------------------------------------------------------
VMMmemoryloadcontrolfacility
Whenaprocessreferencesavirtual-memorypagethatisondisk,becauseiteitherhasbeenpagedoutorhasneverbeenread,thereferencedpagemustbepagedin,andthismaycauseoneormorepagestobepagedoutifthenumberofavailablefreepageframesislow.TheVMMattemptstostealpageframesthathavenotbeenrecentlyreferenced,andthusunlikelytobereferencedinthenearfuture,viathepage-replacementalgorithm.
Asuccessfulpage-replacementkeepsthememorypagesofallcurrentlyactiveprocessesinRAM,whilethememorypagesofinactiveprocessesarepagedout.However,whenRAMisover-committed,itbecomesdifficulttochoosepagesforpageoutbecausetheywillbereferencedinthenearfuturebycurrentlyrunningprocesses.Theresultisthatpagesthatwillsoonbereferencedstillgetpagedoutandthenpagedinagainlater.When
thishappens,continuouspaginginandpagingoutmayoccurifRAMisover-committed.Thisconditioniscalledthrashing.Thesystemspendsmostofitstimepaginginandpagingoutinsteadofexecutingusefulinstructions,andnoneoftheactiveprocessesmakeanysignificantprogress.TheVMMhasamemoryloadcontrolalgorithmthatdetectswhenthesystemisthrashingandthenattemptstocorrectthecondition.------------------------------------------------------------------------
VMSTAT'Savmfield
avmstandsfor"ActiveVirtualMemory"andnot"AvailableMemory".TheavmvalueinVMSTATindicatesthenumberofvirtual-memorypagesthathavebeenaccessedbutnotnecessarilypagedout.Withthepreviouspolicyoflate
pagespaceallocation,avmhadthesamedefinitionbutsincetheVMMwouldallocatepagingspacediskblocksforeachworkingpagethatwasaccessed,thenthepagingspaceblockswereequaltotheavm.Withdeferredpolicy,thepagespacediskblocksareonlyallocatedforthepagesthatneedtobepagedout.Theavmnumberwillgrowasmoreprocessesgetstartedand/orexistingprocessesusemoreworkingstorage.Likewise,thenumberwillshrinkasprocessesexitand/orfreeworkingstorage.------------------------------------------------------------------------
VMSTAT'Sfrefield
freisthenumberof4Kpagesthatarecurrentlyonthefreelist.Whenanapplicationterminates,allofitsworkingpagesareimmediatelyreturned
tothefreelist.Itspersistentpages,however,remaininRAMandarenotaddedbacktothefreelistuntiltheyarestolenbytheVMMforotherprograms.Persistentpagesarealsofreedifthecorrespondingfileisdeleted.
Forthisreason,thefrevaluemaynotindicatealltherealmemorythatcanbereadilyavailableforusebyprocesses.Ifapageframeisneeded,thenpersistentpagesrelatedtoterminatedapplicationsareamongthefirsttobehandedovertoanotherprogram.
-
8/2/2019 Tips for Performance Bottlenecks
24/25
TheminimumnumberofpagesthattheVirtualMemoryManagerkeepsonthefreelistisdeterminedbytheminfreeparameterofvmtune.Ifthenumberofpagesonthefreelistdropsbelowminfree,theVirtualMemoryManagerwillstealpagesuntilthefreelisthasbeenrestoredtothemaxfreevalue.------------------------------------------------------------------------
Howthesystemisusingmemory
Thesvmoncommandcanbeusedtodetermineroughlyhowmuchmemorythesystemisusing.
NOTE:PAIDE/6000mustbeinstalledinordertousesvmon.Checktoseeifthisisinstalled,byexecutingthefollowingcommand:
$lslpp-lperfagent.tools.
IfyouareatAIXVersion4.3.0orhigher,thenthisfilecanbefoundontheAIXBaseOperatingSystemmedia.Otherwise,toorderPAIDE/6000,callIBMDIRECT1-800-426-2255orcontactyourlocalIBMrepresentative.
Asroot,typesvmon.Underthepgspaceheading,theinusefieldisthenumberofworkingpagesthatareinuseinallofvirtualmemory.------------------------------------------------------------------------
Explanationofsvmonoutput
memory:
SIZEtotalsizeofmemoryin4Kpages
INUSEnumberofpagesinRAMthatareinusebyaprocessplusthenumberofpersistentpagesthatbelongedtoaterminatedprocessandarestillresidentinRAM.Thisvalueisthetotalsizeofmemoryminusthenumberofpagesonthefreelist.
FREEnumberofpagesonfreelist.
PINnumberofpagespinnedinRAM(apinnedpageisapagethatisalwaysresidentinRAMandcannotbepagedout)
inuse:
WORKnumberofworkingpagesinRAM
PERSnumberofpersistentpagesinRAM
CLNTnumberofclientpagesinRAM(clientpageisaremotefilepage)
pin:
WORKnumberofworkingpagespinnedinRAM
PERSnumberofpersistentpagespinnedinRAM
CLNTnumberofclientpagespinnedinRAM
pgspace:
SIZEtotalsizeofpagingspacein4Kpages
-
8/2/2019 Tips for Performance Bottlenecks
25/25
INUSEtotalnumberofallocatedslots.(Seeexplanationaboveonallocationofpagingspace).
Tofindouthowmuchmemoryaprocessisusing,type
$svmon-PPID(foroneprocess)or$svmon-Pau|more(forallprocesses)
Toseethenumberofworkingpagesuniquetothisprocess'privatestackanddatauseinallofvirtualmemory,lookattheworktypeanddescriptionprivate.Thesvmonoutputmayalsolistseveralsharedsegments.Foracompletepicture,determinewhichsegmentsareuniquetoanindividualprocessandwhicharesharedwithotherprograms.Multiplythevaluesby4096togetthenumberofbytesinmemorytheprocessisusing.Thenumber4096comesfromthefactthateachpageis4KBinsize.Youcanalsodividethenumberofpagesby256inordertogetmegabytes.