Transcript
Page 1: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Setting up New Relic's Infrastructure: A bunch of things we did wrong and a few we did right.DAVE PETERSON & NIC BENDERS2013-10-25

Wednesday, November 6, 13

Page 3: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

[“star” graph of code base at launch]

Wednesday, November 6, 13

Page 4: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Wednesday, November 6, 13

Page 5: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

100,000req/sec

2Gbps

200TB

Wednesday, November 6, 13

Page 6: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Who Does What?

Wednesday, November 6, 13

Page 7: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Site Operations != IT

Wednesday, November 6, 13

Page 8: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Wednesday, November 6, 13

Page 9: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

You are not Batman

Do your work during the dayWednesday, November 6, 13

Page 10: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Lowest hour is still 60% of peak hour!

Wednesday, November 6, 13

Page 11: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Put your serverson the moon

Wednesday, November 6, 13

Page 12: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Put your serverson the moon

Wednesday, November 6, 13

Page 13: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Never open the box

Wednesday, November 6, 13

Page 14: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Domain names are cheap

Wednesday, November 6, 13

Page 15: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

10.0.0.0/31 10.0.0.0-10.0.0.1 255.255.255.254 210.0.0.0/30 10.0.0.0-10.0.0.3 255.255.255.252 410.0.0.0/29 10.0.0.0-10.0.0.7 255.255.255.248 810.0.0.0/28 10.0.0.0-10.0.0.15 255.255.255.240 1610.0.0.0/27 10.0.0.0-10.0.0.31 255.255.255.224 3210.0.0.0/26 10.0.0.0-10.0.0.63 255.255.255.192 6410.0.0.0/25 10.0.0.0-10.0.0.127 255.255.255.128 12810.0.0.0/24 10.0.0.0-10.0.0.255 255.255.255.0 25610.0.0.0/23 10.0.0.0-10.0.1.255 255.255.254.0 51210.0.0.0/22 10.0.0.0-10.0.3.255 255.255.252.0 102410.0.0.0/21 10.0.0.0-10.0.7.255 255.255.248.0 204810.0.0.0/20 10.0.0.0-10.0.15.255 255.255.240.0 409610.0.0.0/19 10.0.0.0-10.0.31.255 255.255.224.0 819210.0.0.0/18 10.0.0.0-10.0.63.255 255.255.192.0 1638410.0.0.0/17 10.0.0.0-10.0.127.255 255.255.128.0 3276810.0.0.0/16 10.0.0.0-10.0.255.255 255.255.0.0 65536

Private IPsare FREE

Wednesday, November 6, 13

Page 16: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Hardware

Wednesday, November 6, 13

Page 17: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

System Management

Wednesday, November 6, 13

Page 18: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

X6550 E5-2450 E5-2643 E5-2690

Average response time

137

ms

167

ms

158

ms

189

ms

Wednesday, November 6, 13

Page 19: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

25% performance improvement

2x 6-Core X6550@ 2.67 GHz

2x 8-Core E5-2690@ 2.90 GHz

Wednesday, November 6, 13

Page 20: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Storage

Wednesday, November 6, 13

Page 21: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Databases

Wednesday, November 6, 13

Page 22: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

32-bits isn't enoughclass CreateTribbles < ActiveRecord::Migration def change create_table :tribbles do |t| t.boolean :hungry t.string :color t.references :parent, index: true end endend

Wednesday, November 6, 13

Page 23: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

32-bits isn't enough`id` int(11) NOT NULL AUTO_INCREMENT

`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT

`id` SERIAL

Wednesday, November 6, 13

Page 24: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Your database is not a queue

Wednesday, November 6, 13

Page 25: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Long running querieswill kill you

Wednesday, November 6, 13

Page 26: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

And now, a thanks to the people who are out there every day fixing the mistakes that we made:

Wednesday, November 6, 13

Page 27: FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

Jonathan Owens

Jonathan Thurman

Aaron Bento

Jonathan OwensJonathan Owens

Sean Kane

Paul Showalter

Karl Matthias

Wednesday, November 6, 13


Top Related