derek pearcy - reading users' minds for fun and profit
DESCRIPTION
What users say will generally be different from what they do -- this is true, but what's a good strategy when you can't get to enough of your users? What if you could answer some really big questions by performing simple research on ALL of your users? This is the same style of approach taken by companies like Google and Zynga, to target user research efforts which have made them what they are today. Log analysis, done well, can seem like mind-reading. If you haven't done it before: there's nothing to fear.TRANSCRIPT
Log Analysis & Mind Reading
How to influence application development and satisfy customers
Overview
• Rise of the Quants
• Logs: Getting and Handling
• Example Methods
Rise of the Quants: Why?
• Inexpensively optimize what exists
• Target more expensive user research
• Ask big sophisticated questions
Rise of the Quants
Rise of the Quants
• Between June 19 and August 27, 2009: “FarmVille” added on more than 1 million players per week; 11 million daily players.
• Ending “the grind”: Rewards and Pacing
Rise of the Quants: WoW
Logs: Getting & Handling
• Getting
• Handling
• Legal concerns
Consider what you want
Consider what you want
Find out what you’ve got
Find out what you’ve got
AnalyzeAnalyze
Grow thesample setGrow the
sample set
Getting
• Where are the actual logs?
• Consider what you want
• Make friends with customer service
• …with field reps
• Change the process
• Change the application
hours
months
Getting
• Read logs
• Don’t take anyone’s word for anything
• Get your own copy of the application
Handling
• Normalization
• Analysis
• Supplementation
Normalization
• Enforce consistency
• Indexing for uniqueness
Analysis
• First-pass tools
• Second-pass tools
Example Methods
Example Methods: simple
[18/Sep/2008:14:11:13 +0000] [CLUSTER HTTPS] utacs wdNbTawxTApoSY8 10.100.144.21 "login" "" "" "" [18/Sep/2008:14:11:13 +0000] [CLUSTER HTTPS] utacs wdNbTawxTApoSY8 10.100.144.21 "setting added" "UA" WhiteList.ListEntries[100] "-NA-" "www.historyteacher.org,8,11,'<Enter%20description%20here>'," /conf?navTo=URLWhite[18/Sep/2008:14:11:14 +0000] [CLUSTER HTTPS] utacs fFVnSzCyKdxVQSR 10.100.144.21 "login" "" "" "" [18/Sep/2008:14:49:30 +0000] [HTTPS] Admin TDto4a0iCBuoQdr 10.100.12.129 "logout (session timeout)" "" "" "" [18/Sep/2008:15:33:59 +0000] [CLUSTER HTTPS] schis E0UtufI3k8pqwDY 10.100.144.21 "login" "" "" "" [18/Sep/2008:15:33:59 +0000] [CLUSTER HTTPS] schis E0UtufI3k8pqwDY 10.100.144.21 "setting added" "byUsername" WhiteList.ListEntries[30] "-NA-" "http://www.techwebonlineevents.com/,8192,11,'Information%20Week%20Webinar%20Registrations',*" /conf?navTo=URLWhite[18/Sep/2008:15:44:47 +0000] [CLUSTER HTTPS] bcadmin TlpuUatuzwnl4XI 10.100.144.21 "login" "" "" "" [18/Sep/2008:15:44:47 +0000] [CLUSTER HTTPS] bcadmin TlpuUatuzwnl4XI 10.100.144.21 "setting added" "byUsername" WhiteList.ListEntries[24] "-NA-" "http://www.furnbrain.com,8,11,'Educational%20site%20requested%20from%20Mrs.%20Kramer'," /conf?navTo=URLWhite[18/Sep/2008:15:53:42 +0000] [CLUSTER HTTPS] bcadmin +g4WxtRjBTOJIFO 10.100.144.21 "login" "" "" "" [18/Sep/2008:15:54:01 +0000] [CLUSTER HTTPS] bcadmin 8YbKF5X/bdlw8eR 10.100.144.21 "login" "" "" "" [18/Sep/2008:15:54:02 +0000] [CLUSTER HTTPS] bcadmin 8YbKF5X/bdlw8eR 10.100.144.21 "setting changed" "byUsername" WhiteList.ListEntries[24] "http://www.furnbrain.com,8,11,'Educational%20site%20requested%20from%20Mrs.%20Kramer'," "www.furnbrain.com,8,11,'Educational%20site%20requested%20from%20Mrs.%20Kramer'," /conf?navTo=URLWhite[18/Sep/2008:15:54:27 +0000] [CLUSTER HTTPS] bcadmin I7SWNNrv8QSTeNW 10.100.144.21 "login" "" "" "" [18/Sep/2008:15:54:27 +0000] [CLUSTER HTTPS] bcadmin I7SWNNrv8QSTeNW 10.100.144.21 "setting changed" "byUsername" WhiteList.ListEntries[24] "www.furnbrain.com,8,11,'Educational%20site%20requested%20from%20Mrs.%20Kraner'," "www.funbrain.com,8,11,'Educational%20site%20requested%20from%20Mrs.%20Kramer'," /conf?navTo=URLWhite[18/Sep/2008:16:02:45 +0000] [CLUSTER HTTPS] Admin MyojZGN9/Nqu6Oq 10.100.144.21 "login" "" "" "" [18/Sep/2008:16:02:45 +0000] [CLUSTER HTTPS] Admin MyojZGN9/Nqu6Oq 10.100.144.21 "setting deleted" "MEGguest" AccessControl.BlockList[0] ""ebay.com";"<Enter%20description%20here>";*#" "-NA-" /conf?navTo=FilterByExpressions
"setting deleted" "MEGguest" AccessControl.BlockList[0] ""ebay.com";"[...]";*#" "-NA-" /conf?navTo=FilterByExpressions
"setting added" "byUsername" WhiteList.ListEntries[30] "-NA-" "http://www.techweb...,*" /conf?navTo=URLWhite
Example Methods: simple
"setting deleted"
"MEGguest"AccessControl.BlockList[0]
""ebay.com";"[...]";*#"
"-NA-"
/conf?navTo=FilterByExpressions
"setting added" "byUsername”WhiteList.ListEntries[30]
"http://www.techweb...,*"
"-NA-"/conf?navTo=URLWhite
setting deleted "MEGguest"AccessControl.BlockList[0]
ebay.com "-NA-"
/conf?navTo=FilterByExpressions
setting added "byUsername”WhiteList.ListEntries[30]
http:// techweb.com
"-NA-"/conf?navTo=URLWhite
before
after
Example Methods: simple
2010-04-12 20:58:05.493 analysisApp[19526:207] |setting added| HTTPSProxy.TrustedCAS[85] |CertificateVerification2010-04-12 20:58:05.494 analysisApp[19526:207] |setting added| HTTPSProxy.TrustedCAS[15] |CertificateVerification2010-04-12 20:58:05.494 analysisApp[19526:207] |manually triggered crl update| -NA- |CertificateRevocationLists2010-04-12 20:58:05.495 analysisApp[19526:207] |setting deleted| SSLScanner.CertificateList[1] |CertificateList2010-04-12 20:58:05.495 analysisApp[19526:207] |setting added| HTTPProxy.ListenerPorts[1] |HTTPProxySettings2010-04-12 20:58:05.496 analysisApp[19526:207] |setting deleted| HTTPProxy.ListenerPorts[1] |HTTPProxySettings2010-04-12 20:58:05.496 analysisApp[19526:207] |setting added| SSLScanner.CertificateList[1] |CertificateList2010-04-12 20:58:05.496 analysisApp[19526:207] |setting added| SSLScanner.CertificateList[2] |CertificateList2010-04-12 20:58:05.497 analysisApp[19526:207] |setting deleted| SSLScanner.CertificateList[2] |CertificateList2010-04-12 20:58:05.497 analysisApp[19526:207] |manually triggered crl update| -NA- |CertificateRevocationLists2010-04-12 20:58:05.498 analysisApp[19526:207] |setting changed| SSLScanner.CertificateList[1] |CertificateList2010-04-12 20:58:05.498 analysisApp[19526:207] |setting changed| SSLScanner.CertificateList[1] |CertificateList
Example Methods: complex
default.conf
default.conf
“factory default”
“customer’s default”
westcoast.conf students.conf POTUS.conf
“customer’s customized”
Note: I am not a programmer. Be patient, think about what you want to do, look at examples.
In this case, correlation nearly aligned with number of values per row.
Thanks.
Not formally approved by
Opinions are speaker’s own
Credits
• “Hang in there, baby,” by binkle_28, via Flickr.
• “Artsy Stambul,” by maistora, via Flickr.
• “Old tools,” by Svadilfari, via Flickr.
• Black-and-white photo, “Tool trader,” by Meena Kadri, also via Flickr.
• Other materials © McAfee, Inc. or their copyright holder. All rights reserved.