building internet-scale applications

Building Internet-scale Applications:The Beginning

By @gaveen At #RefreshColombo 22 September 2011

Gaveen Prabhasara

twitter.com/gaveen

SysAdmin

DevOps

Works at Thinkcubegaveen.owain.org

github.com/gaveen

gaveens@gmail.com

Standing on theShoulders of Giants

If I claim to be a wise man, wellIt surely means that I don't know

“

“

- Kansas Carry On Wayward Son

This is a 25 minute talk

Which sounds like a collection of

Random thoughts about

Building Internet-scale Systems

You probably already know

Internet-scale

What is it?

Applications

Does not always mean WebApps

Internet-scale > Enterprise-scale

Facebook,YouTube, Twitter,Gmail, Flikr

Eg:

Case: Facebook750M+ active users350M+ daily users100B+ daily hits30B+ monthly objects shared2.5M+ websites integrated130TB+ daily logs*Sources: http://goo.gl/zT1bL

http://goo.gl/cdFPI

Case: Akamai95800+ servers1000+ networks70+ countries15-30% WWW traffic*Source: http://goo.gl/yVmG9

Culture

Culture

Culture

How

Know Your Trade

Learn

Acquire domain-specific knowledge

Learn how people build similar services

Learn about Technologies

Stay updated

RTFM / Use Search Engines ;)

Get Right People

Team

Find balance in the team

Know their strengths

Nurture them

Set a proper environment

Don't step on others toes

Go for the Best Solution

Architecture & Design

Have a solid architecture

Find a good engineering design

Don't over-engineer / over-complicate

Avoid architecture degradation

Right Tool for the Right Job

Technology

You favourite tool can't do everything. Face it

'Prejudice against' & 'religious for' isn't good for you

Brand names aren't for worshiping

Cloud isn't a magical silver bullet

This is Spar... the Internet. It's big

Do Things Right

Process

Use best practices

Automate. Automate. Automate.

Plan for scaling from the start

Don't let the process be a burden

Little more detail...

Test All the Time or better,do Continuous Integration

Little more detail...

Testing isn't a QA job

Let Devs take responsibility

XP, TDD, BDD, Cucumber, etc.

Don't push untested code

Works-in-my-machine doesn't count

Process

Fight the Smell, Fight Anti-Patterns

Little more detail...

Re-factoring is good

Code smell isn't something to behold

Don't let the code age on you

Don't let the architecture degrade

Bad code is bad code

Process

Use Version Control

Little more detail...

Use a proper VCS

Git, SVN, Hg, Bzr, etc.

Helps automation

No version control, no deploying

Learn to use a VCS properly

Process

Use Configuration Management

Little more detail...

Infrastructure is Code, therefore Automate

User a Configuration Management System

Puppet, Chef, CFEngine, MCollective, etc.

One-off work is (usually) a myth

Managed Infra opposed to mucked-about infra

DevOpsProcess

Security isn't a Task

Little more detail...

You can't apply security later

Both infra & code needs to be secure

“Noboday wants to hack me” is a myth

HTTPS isn't a fad

# chmod 777 kills puppies

SELinux / AppArmor isn't your enemy

Process

Scaling starts from the beginning

Little more detail...

Plan early. Start from basics.

Balance load & plan for HA

Use Caching

Do stress / load testing

Not every technology scales gracefully

Use messaging, async jobs when needed

Etc., etc.

Process

Monitor & Measure

Little more detail...

Capture important metrics

Measure, analyze & improve

Monitoring sucks. But still...

Home grown a NMS aren't necessarily better

Have notification alerting in place

Process

Q & A

Anybody who wants ISO's of the following Linux distros,meet me with USB drives after the session.

BackTrack, CentOS, Debian, Fedora, Hanthana, MeeGo, Mint,Ubuntu, Ubuntu Server