a linux and/or vm solution for analyzing any web server log

A A Linux Linux andor VM Solution forandor VM Solution forAnalyzing any Web Server LogAnalyzing any Web Server Log

SHARE 99 Session 9108 August 2002SHARE 99 Session 9108 August 2002

Gordon W Gordon W WolfeWolfe PhPhDD

Senior VM Systems ProgrammerSenior VM Systems Programmer

VM Technical Services Shared Systems Group The Boeing CompanyVM Technical Services Shared Systems Group The Boeing Company

e-mail e-mail gordongordonwwwolfewolfeboeingboeingcomcom

Whorsquos Looking At YourWhorsquos Looking At YourWebserverWebserver

bullbull Is your Is your website website effectiveeffective

bullbull Does anyone look at itDoes anyone look at it

bullbull WHOrsquos looking at it Where are theyWHOrsquos looking at it Where are they

bullbull Do they come direct or are they referred by anotherDo they come direct or are they referred by anothersitesite

bullbull WHAT do they look atWHAT do they look at

bullbull How many hits are you gettingHow many hits are you getting

The Problem at BoeingThe Problem at Boeing

bullbull CA VMCA VMWebserver Webserver sites have over 1 million hits persites have over 1 million hits permonthmonth

bullbull Unix and NT servers have logs too big to analyzeUnix and NT servers have logs too big to analyzelocally want to run on mainframelocally want to run on mainframe

bullbull How do we find out whatrsquos the most effective use ofHow do we find out whatrsquos the most effective use ofhardware and softwarehardware and software

bullbull Tried writing our own analysis programbut must beTried writing our own analysis programbut must becustomized for every servercustomized for every server

Stephen Turnerrsquos ANALOGStephen Turnerrsquos ANALOGProgramProgrambullbull Someonersquos already written this programSomeonersquos already written this program

bullbull Stephen Turner University of Cambridge (UK) StatisticalStephen Turner University of Cambridge (UK) StatisticalLaboratoryLaboratory

bullbull The program is FREEThe program is FREE

bullbull TheThe standard analysis program - runs on 90 of analysis sites standard analysis program - runs on 90 of analysis sites

bullbull Extremely flexible - will analyze virtually all Extremely flexible - will analyze virtually all webserverwebserver log logformats with APACHE and NT built informats with APACHE and NT built in

bullbull Written in C runs on Unix Windows Macintosh zOS DECWritten in C runs on Unix Windows Macintosh zOS DECAS400 OS2 BUT NOT VM or AS400 OS2 BUT NOT VM or LinuxLinux

Porting ANALOG 40 toPorting ANALOG 40 toVMVMOpenExtensionOpenExtensionbullbull This author took Unix version and ported to VM This author took Unix version and ported to VM OpenExtensionOpenExtension Simpler Simpler

than rewriting whole thing to use CMS file structuresthan rewriting whole thing to use CMS file structures

bullbull Used Used NealeNeale Fergusonrsquos Fergusonrsquos ldquoPorting UNIX Applications to ldquoPorting UNIX Applications to OpenEditionOpenEdition for forVMESArdquo VMESArdquo redbookredbook SG24-4747-00 SG24-4747-00

bullbull Port was not simple even though program already had provision forPort was not simple even though program already had provision forEBCDICEBCDIC

ndashndash Rewrote Rewrote MakefileMakefile

ndashndash Migrate LE370 H files into usrlibinclude Resolved conflicts with TCPIP HMigrate LE370 H files into usrlibinclude Resolved conflicts with TCPIP Hfiles of same namefiles of same name

ndashndash Rewrite 3 of Turnerrsquos H files and one subroutine for VMCMSRewrite 3 of Turnerrsquos H files and one subroutine for VMCMS

ndashndash MrMr Turner had one small bug in his code that only affected VMCMS Turner had one small bug in his code that only affected VMCMS

ndashndash Took 4 months to do the portTook 4 months to do the port

ndashndash Complete compile and link-edit takes 40 minutes on R85Complete compile and link-edit takes 40 minutes on R85

bullbull Currently working on port for Analog 50Currently working on port for Analog 50

Porting ANALOG 50 toPorting ANALOG 50 toLinuxLinux390390

bullbull LinuxLinux390 port was simple390 port was simple

bullbull Took UNIX version of ANALOG copied toTook UNIX version of ANALOG copied toLinuxLinux CompliledCompliled linked and ran clean the linked and ran clean thefirst timefirst time

Running from OERunning from OE

bullbull Put ANALOG program in usrlocalbinPut ANALOG program in usrlocalbin

bullbull Put run-time files in usrlocalAnalog40Put run-time files in usrlocalAnalog40

ndashndash UsdomUsdomtabtab

ndashndash UsUslnglng

bullbull Put work files in Put work files in tmp tmp or on CMSor on CMS

ndashndash DnsDnsfilefile

bullbull Put analogPut analogcfg cfg in user directory homeuserin user directory homeuser

ndashndash Config Config file is the key to running Analogfile is the key to running Analog

ndashndash Tells where the data isTells where the data is

ndashndash Tells where the run-time files areTells where the run-time files are

ndashndash Tells how you want the data analyzedTells how you want the data analyzed

bullbull Run by entering analog +g homeuseranalogRun by entering analog +g homeuseranalogcfgcfg

Running from CMS DirectlyRunning from CMS Directly

bullbull Takes advantage of the fact that you can run OETakes advantage of the fact that you can run OEdirectly from CMS with OPENVM EXECdirectly from CMS with OPENVM EXEC

bullbull Create an ANALOG EXEC to run it place on 19E diskCreate an ANALOG EXEC to run it place on 19E disk(See Appendix)(See Appendix)

bullbull OE programs can call CMS files by simply using theOE programs can call CMS files by simply using thestructure filenamestructure filenamefiletypefiletypefilemodefilemode

bullbull Run using EXEC ANALOG Run using EXEC ANALOG configfn configft configfmconfigfn configft configfm

bullbull 350000 records analyzed in 9 seconds350000 records analyzed in 9 seconds

Remote JobsRemote Jobs

bullbull Use CArsquos VMBatchrsquos RJE facilityUse CArsquos VMBatchrsquos RJE facility

bullbull ANALOG EXEC on 19E diskANALOG EXEC on 19E disk

bullbull For accounting set up dummy For accounting set up dummy useridsuserids

ndashndash USER REMOTE1 XXX G - perhaps same as node name of remote serverUSER REMOTE1 XXX G - perhaps same as node name of remote server

ndashndash ACCOUNT CHARGE1ACCOUNT CHARGE1

ndashndash No IPL card or MDISK cards - charge only for CPU timeNo IPL card or MDISK cards - charge only for CPU time

bullbull Then send job deck to VMBATCH in RJE formatThen send job deck to VMBATCH in RJE format

ndashndash Ist Ist card RJECMD USER REMOTE1 PASS XXX NODE REMOTE1card RJECMD USER REMOTE1 PASS XXX NODE REMOTE1

ndashndash 2nd card EXEC ANALOG ANALOG CONFIG A2nd card EXEC ANALOG ANALOG CONFIG A

ndashndash 3rd card ENDCMD3rd card ENDCMD

ndashndash 4th card READ ANALOG CONFIG A4th card READ ANALOG CONFIG A

ndashndash Add cards for above fileAdd cards for above file

ndashndash 5th card READ ANALOG DATA A5th card READ ANALOG DATA A


bullbull Send whole deck in NETDATA formatSend whole deck in NETDATA format

Running ANALOG on Running ANALOG on LinuxLinux

bullbull Binary executable usrbinanalogBinary executable usrbinanalog

bullbull Place files in usrPlace files in usrsrcsrc usr usrsrcsrcanalog-501analog-501langlangusdomusdomtab and usrtab and usrsrcsrcanalog-analog-501501langlangususlnglng Also Also analogoanalogogif gif to analog to analog subdir subdir of html subdirectory ofof html subdirectory ofAPACHEAPACHE

bullbull Dummy control file usrbinanalogDummy control file usrbinanalogcfg cfg just has one entry just has one entry CONFIGFILECONFIGFILEetcanalogetcanalogcfg cfg to point to real to point to real config config file in rw subdirectory etcfile in rw subdirectory etc

bullbull Point etcanalogPoint etcanalogcfg cfg entries to where you have (or want) filesentries to where you have (or want) files

ndashndash LOGFILE LOGFILE varvarlogloghttpdhttpdaccess_log lt==the log file to be analyzedaccess_log lt==the log file to be analyzed

ndashndash OUTFILE homeOUTFILE homehttpdhttpdhtmlanaloghtml lt==where you want the output (for APACHE)htmlanaloghtml lt==where you want the output (for APACHE)

ndashndash IMAGEDIR analog lt== output relative to above directory eg homeIMAGEDIR analog lt== output relative to above directory eg homehttpdhttpdhtmlanaloghtmlanalog

ndashndash DOMAINSFILE usrDOMAINSFILE usrsrcsrcanalog-501analog-501langlangusdomusdomtabtab

ndashndash LANGFILE usrLANGFILE usrsrcsrcanalog-501analog-501langlangususlnglng

ndashndash DNSFILE DNSFILE tmptmpdnsdnsfile lt== Temporary DNS lookup file created by analogfile lt== Temporary DNS lookup file created by analog

ndashndash DNSLOCKFILE DNSLOCKFILE tmptmpdnslock dnslock lt== Temp lock file created by analog lt== Temp lock file created by analog

bullbull Then to run just enter usrbinanalogThen to run just enter usrbinanalog

TheThe Config Config File - Key to File - Key toANALOG processingANALOG processingbullbull LOGFORMAT hellip to inform analog of format of logsLOGFORMAT hellip to inform analog of format of logs

bullbull LOGFILE tells analog name of log file to analyzeLOGFILE tells analog name of log file to analyze

bullbull OUTFILE tells analog where to put results Output is report or direct toOUTFILE tells analog where to put results Output is report or direct toHTML for web displayHTML for web display

bullbull DOMAINSFILE - where the table of domains isDOMAINSFILE - where the table of domains is

bullbull DNSFILE tells analog where to put the DNSFILE tells analog where to put the dns dns cache filecache file

bullbull All above are minimum requirements to run AnalogAll above are minimum requirements to run Analog

bullbull Other entries to specify what reports you want what data to include andOther entries to specify what reports you want what data to include andexclude what certain file extensions mean and so onexclude what certain file extensions mean and so on

LOGFORMAT valuesLOGFORMAT values

bullbull Analogrsquos default is CERNNCSA common format which is usedAnalogrsquos default is CERNNCSA common format which is usedby NCSA server and APACHEby NCSA server and APACHE

bullbull Has built-in values for Windows (9598NT) WEBSTAR andHas built-in values for Windows (9598NT) WEBSTAR andNETSCAPE formatsNETSCAPE formats

bullbull Anything else requires custom formatAnything else requires custom format

ndashndash Analog is really picky about the formatAnalog is really picky about the format

ndashndash Must match the format of the log exactlyMust match the format of the log exactly

bullbull For VMFor VMWebserverWebserver use the VIWLOGEX utility to convert log files use the VIWLOGEX utility to convert log filesto CERN formatto CERN format

Sample Results 1Sample Results 1





Where Can I Get AnalogWhere Can I Get Analog

bullbull CMS version 40 from this author or at CMS version 40 from this author or at Neale Fergusonrsquos Neale Fergusonrsquos OE siteOE sitehttphttppuccpuccprincetonprincetoneduedu~~nealenealevmoe03htmlANALOG or fromvmoe03htmlANALOG or fromthis authorthis author

bullbull LinuxLinux390 version 50 from http390 version 50 from httplinuxvmlinuxvmorgInfoanalg501orgInfoanalg501tgztgz ororthis author only at this timethis author only at this time

bullbull Source code at Stephen Turnerrsquos site Source code at Stephen Turnerrsquos site httpwwwanaloghttpwwwanalogcxcx

bullbull Complete on-line manual for ANALOG also at Turnerrsquos siteComplete on-line manual for ANALOG also at Turnerrsquos site

ANALOG ANALOG ListserverListserver

bullbull Have questions or problems with ANALOGHave questions or problems with ANALOG

bullbull Try the ANALOG Try the ANALOG listserverlistserver

bullbull Questions answered by people whorsquove had the sameQuestions answered by people whorsquove had the sameproblemsproblems

bullbull Frequently will be answered by Stephen TurnerFrequently will be answered by Stephen Turnerhimselfhimself

bullbull httpwwwanaloghttpwwwanalogcxcxdocsmailinghtmldocsmailinghtml

Analog 50Analog 50

bullbull Minor new featuresMinor new featurescommands and bugcommands and bugfixesfixes

bullbull Big news is pie chartsBig news is pie chartsand and JPEGsJPEGs

Appendix SampleAppendix Sample Configfile Configfile for forVMVMWebserverWebserver

LOGFORMAT COMBINED LOGFILE SCM1CERN0106C HOSTNAME Boeing VMWebserver LANGFILE usrlocalanalog40uslng OUTFILE SCM10106HTMLF DOMAINSFILE usrlocalanalog40ukdomtab OUTPUT HTML DNSFILE dnsfile DNS WRITE DNSGOODHOURS 800 IMAGEDIR images GOTOS OFF REQINCLUDE pages LINKINCLUDE pages REFLINKINCLUDE pages UNCOMPRESS gzZ gzip -cd BROWOUTPUTALIAS Mozilla Netscape BROWOUTPUTALIAS Mozilla (compatible) Netscape (compatible) BROWOUTPUTALIAS IWENG AOLSUBTYPE gzZ TYPEOUTPUTALIAS html html [Hypertext Markup Language] TYPEOUTPUTALIAS htm htm [Hypertext Markup Language] TYPEOUTPUTALIAS ps ps [PostScript] TYPEOUTPUTALIAS gz gz [Gzip compressed files]

Appendix SampleAppendix Sample Configfile Configfile for forLinuxLinux Apache Apache

LOGFILE varloghttpdaccess_logOUTFILE homehttpdhtmlanaloghtmlDOMAINSFILE usrsrcanalog-501langusdomtabLANGFILE usrsrcanalog-501languslngHOSTNAME Linux390 Apache WebserverOUTPUT HTMLDNSFILE tmpdnsfileDNSLOCKFILE tmpdnslockDNS WRITEDNSGOODHOURS 800 following directory is relative to homehttpdhtmlIMAGEDIR analogGOTOS OFFLINKINCLUDE pagesREQLINKINCLUDE pagesREFLINKINCLUDE REDIRREFLINKINCLUDE FAILREFLINKINCLUDE OSREP ONUNCOMPRESS gzZ gzip -cdBROWOUTPUTALIAS Mozilla NetscapeBROWOUTPUTALIAS Mozilla (compatible) Netscape (compatible)BROWOUTPUTALIAS IWENG AOLSUBBROW SUBTYPE gzZ Add whichever of these types of pages you have on your server or othersPAGEINCLUDE shtmlPAGEINCLUDE aspPAGEINCLUDE jsp

Appendix Sample EXEC forAppendix Sample EXEC forCMSCMS

EXEC to run Analog 40 from CMS EXEC ANALOG ltconfig filegt the config file should reside on your a-disk by Gordon Wolfe VM Technical Services 031600 address command arg cfgfn cfgft cfgfm EXEC OPENVM MOUNT VMBFSVMSYSROOT EXEC OPENVM SET DIRECTORY homewebstat r = rc if r ltgt 0 then do say Cannot mount BFS exit r end EXEC OPENVM RUN usrlocalbinanalog +gcfgfncfgftcfgfm exit rc

Whorsquos Looking At YourWhorsquos Looking At YourWebserverWebserver

bullbull Is your Is your website website effectiveeffective

bullbull Does anyone look at itDoes anyone look at it

bullbull WHOrsquos looking at it Where are theyWHOrsquos looking at it Where are they

bullbull Do they come direct or are they referred by anotherDo they come direct or are they referred by anothersitesite

bullbull WHAT do they look atWHAT do they look at

bullbull How many hits are you gettingHow many hits are you getting








































































































Analog 50Analog 50
















































































































Analog 50Analog 50









a linux and/or vm solution for analyzing any web server log

Documents