Software Instability

Renato Lucindo@rlucindo -- lucindo.github.com

FFFFUUUU Conf 2010 -- ffffuuuu.me

What kind of software?

Web ApplicationsKinda BigDistributed Systems

Production-ready Software

worksstablemaintainabledocumentedscalable

Real life Production Software

it runs

Anatomy of a Real life Software

focus on featurespoor logsno administrative toolsno informationpoorly tested

Why?

Feature centric"What're the odds of that happening?"DeadlinesReReReReReReprioritization

Over confidenceInexperienceNaivenessOver excitement

The Clients...

Product Owner?Manager?Stakeholder?

Who will deal with your mess?

OPERATIONS

Building and Running a System

What to do?

Non-functional Requirements

decent logswith context (user, host, operation, ...)

automatic reload (configs, templates, ...)administrative tools

plenty of itcounters and statisticschartsoperations requirements

Tests that matter

Fully automated functional testNo mocksLog replayHigh concurrency testBullet-proof tests

The -STOP testLoad test (iozone, gcc test)Kill test

How to add a new feature

On/Off switchTests with different versionsFull system functional testsUpgrade testDowngrade (rollback) test

Architecture 101

Cache

Queue

Partition

Stability Patterns

Use TimeoutsCircuit BreakerBulkheadsSteady StateFail FastHandshakingTest HarnessDecoupling Middleware

Use your head!

What to expect

CAP Theorem

Failures

Hardware failureNetwork failureElectricityNatural disastersCode failure

Bugs

Human Errors

Your cluelessness