a node.js bag of goodies for analyzing web traffic
TRANSCRIPT
![Page 1: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/1.jpg)
A Node.JS Bag of Goodies for Analysing WebTraffic
Philip Tellis / [email protected]
ConFoo / 2012-03-02
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 1
![Page 2: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/2.jpg)
Leave feeback on this talk at joind.in/5967
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 2
![Page 3: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/3.jpg)
$ finger philip
Philip [email protected]
@bluesmoongeek - paranoid - speedfreakco-founder @ Log-Normalhttp://bluesmoon.info/
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 3
![Page 4: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/4.jpg)
Log-Normal
We do real user web performance analysis
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 4
![Page 5: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/5.jpg)
Log-Normal
This talk covers some of the Node.JS modules we use to do thisanalysis
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 5
![Page 6: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/6.jpg)
Examples for this talk
http://bluesmoon.github.com/talks/node-modules/
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 6
![Page 7: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/7.jpg)
–Node.JS & npm
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 7
![Page 8: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/8.jpg)
0.1 The 2 minute Node.JS Tutorial
node> m = require(’module’)> console.log(m.foo)
We’ll be using node v0.6.x
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 8
![Page 9: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/9.jpg)
0.1 The 2 minute Node.JS Tutorial
node script.js
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 9
![Page 10: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/10.jpg)
0.1 The 2 minute Node.JS Tutorial
echo "console.log(’hello, world!’)" | node
Notice that in JavaScript, semicolons are optional
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 10
![Page 11: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/11.jpg)
0.2 Installing modules
npm install module
Installs into a local node_modules directory
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 11
![Page 12: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/12.jpg)
0.2 Installing modules globally
sudo npm install -g module
Installs into the /usr/lib/node_modules directory
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 12
![Page 13: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/13.jpg)
0.2 npm help
npm help npm
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 13
![Page 14: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/14.jpg)
0.3 npmjs.org
search.npmjs.org
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 14
![Page 15: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/15.jpg)
What’s the time?
13:37?
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 15
![Page 16: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/16.jpg)
1HTTP Logs
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 16
![Page 17: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/17.jpg)
1.1 UserAgent Parsing
npm install ua-parser
var parser=require(’ua-parser’);var ua = parser.parse(ua_string);
// family, major, minor, patch, os
Extracted from Google’s BrowserScope Project
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 17
![Page 18: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/18.jpg)
1.2 IP Addresses
var net = require(’net’);
net.isIP(ip); // returns 0, 4 or 6
The net module is part of node
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 18
![Page 19: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/19.jpg)
1.3 GeoLocation
npm install geoip-lite
var geo = require(’geip-lite’);var loc = geo.lookup(ip);
// country, region, city, ll
Uses MaxMind’s GeoIP database. Very fast lookups, IPv4 & v6
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 19
![Page 20: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/20.jpg)
1.3 geoip-lite’s hidden function
var geo = require(’geip-lite’);geo.cmp(ip1, ip2);
// -1, 0 or 1
Used internally to do a binary search on the IP database
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 20
![Page 21: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/21.jpg)
1.4 Date formatting
npm install prettydate
var strftime = require(’prettydate’).strftime;
var dstr = strftime(new Date, "%c");
Also accepts a locale as a third parameter.
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 21
![Page 22: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/22.jpg)
1.5 Querystring Parsing
var qs = require(’querystring’);
qs.parse(’name=Larry&name=Moe&name=Curly’);// { name: [ ’Larry’, ’Moe’, ’Curly’ ] }
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 22
![Page 23: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/23.jpg)
1.6 Creating Hashes
var crypto = require(’crypto’);
var hash = crypto.createHash(’sha1’);hash.update(data);
var digest = hash.digest(’hex’);
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 23
![Page 24: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/24.jpg)
1.6 Creating HMACs
var crypto = require(’crypto’);
var hmac = crypto.createHmac(’sha1’, key);hmac.update(data);
var digest = hmac.digest(’base64’);
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 24
![Page 25: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/25.jpg)
What’s the time?
It should be14:05I hope ;)
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 25
![Page 26: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/26.jpg)
2Statistical Analysis
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 26
![Page 27: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/27.jpg)
2.1 faststats
npm install fast-stats
var Stats = require(’fast-stats’).Stats;var s = new Stats().push(1, 2, 3, 10, 8, 4, 3);console.log(s.amean().toFixed(2));
Caveat: I haven’t pushed out a new version in a while
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 27
![Page 28: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/28.jpg)
2.2 gauss
npm install gauss
var gauss = require(’gauss’);var set = new gauss.Vector(5, 1, 3, 2, 21);console.log(set.mean());
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 28
![Page 29: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/29.jpg)
2.3 statsd
npm install statsd node-statsd
github.com/etsy/statsdEasy to set upRequires graphite to plot chartsBrought to you by Etsy
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 29
![Page 30: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/30.jpg)
2.3 statsd
var StatsD = require(’node-statsd’).StatsDc = new StatsD(’example.org’,8125)c.timing(’node_test.some_service.task.time’, 500)
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 30
![Page 31: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/31.jpg)
2.4 cube
Disclaimer: I’ve never used this one before
npm install cube
square.github.com/cube/github.com/square/cube/wiki
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 31
![Page 32: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/32.jpg)
What’s the time?
14:20?
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 32
![Page 33: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/33.jpg)
Help me out
Still trying to figure out the best way to debug Node.JS memoryusage. Ideas? Let me know.
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 33
![Page 34: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/34.jpg)
Questions?
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 34
![Page 35: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/35.jpg)
Leave feeback on this talk at joind.in/5967
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 35
![Page 36: A Node.JS bag of goodies for analyzing Web Traffic](https://reader031.vdocuments.us/reader031/viewer/2022013118/5557d464d8b42ab6258b4903/html5/thumbnails/36.jpg)
Contact me
Philip [email protected]
@bluesmoongeek - paranoid - speedfreakco-founder @ Log-Normalhttp://bluesmoon.info/slideshare.net/bluesmoonjoind.in/5967
ConFoo / 2012-03-02 A Node.JS Bag of Goodies for Analysing Web Traffic 36