why and how to test logging - devops showcase north - feb 2016 - matthew skelton

Why and how to test loggingDevOps Showcase North / Testing Showcase North

Manchester, UK - UnicomThursday 11th February 2015

Matthew SkeltonSkelton Thatcher Consulting

confession:

I am a big fan of logging

use logging as a channel/vector to make distributed

systems more testable

transforming technology and teams

high impact expertise

Cloud, Agile, DevOps

Why we should test logging

How to test logging effectively

exceptional situationsedge cases

metricsanalytics‘audits’

…@evanphx

logging as an execution trace

reduce time-to-detectincrease team engagement

increase configurabilityenhance DevOps collaboration

#operability

(event sourcing)

(structured logging)

(CQRS)

using logging mainly for errors

inconsistent use of logging

“logging slows things down”

“logging pollutes my domain model”

“logging is just for Ops people”

log aggregation only in Production

logging assumed to be free ($0) to implement

logs not available to Tester or Developers

distinct event IDs

transaction tracing

instrumentation

log aggregation & search

cross-component visibility

increased test coverage

programmatic assertions

deep insights

stories for testing logging

BasketItemAdded

Search: BasketItem

Continuous event IDs

How many distinct event types (state transitions) in

your application?

represent distinct states

enum

Human-readable sets: unique values, sparse, immutable

C#, Java, Python, node(Ruby, PHP, …)

public enum EventID

{

// Badly-initialised logging data

NotSet = 0,

// An unrecognised event has occurred

UnexpectedError = 10000,

ApplicationStarted = 20000,

ApplicationShutdownNoticeReceived = 20001,

PageGenerationStarted = 30000,

PageGenerationCompleted = 30001,

MessageQueued = 40000,

MessagePeeked = 40001,

BasketItemAdded = 60001,

BasketItemRemoved = 60002,

CreditCardDetailsSubmitted = 70001,

// ...

}

Technical

Domain

public enum EventID

{

// Badly-initialised logging data

NotSet = 0,

// An unrecognised event has occurred

UnexpectedError = 10000,

ApplicationStarted = 20000,

ApplicationShutdownNoticeReceived = 20001,

PageGenerationStarted = 30000,

PageGenerationCompleted = 30001,

MessageQueued = 40000,

MessagePeeked = 40001,

BasketItemAdded = 60001,

BasketItemRemoved = 60002,

CreditCardDetailsSubmitted = 70001,

// ...

}

BasketItemAdded = 60001

BasketItemAdded = 60001

BasketItemRemoved = 60002

BasketItemAdded = 60001

BasketItemRemoved = 60002

represent distinct states

OrderSvc_BasketItemAdded

Monolith to microservices:debugger does not have the full view

Even with remote debugger, it’s boring to attach and detach

Storage I/O

Worker Job

Queue

Upload

Transaction tracing

‘Unique-ish’ identifier for each request

Passed through downstream layers

Unique-ish ID

Testing logging

use logging as a channel/vector to make distributed

systems more testable

track a parcel online

search by event ID

transaction trace

Web App

Doc store

Audit service

Msg queue

Transactional sub-system

Log aggregation sub-system

An action here… …results in log entry here

trace events in a distributed,

asynchronous system

stories for testing logging

AuditRecordCreated

Search: AuditRecordCreated

AuditRecordCreated

Search: AuditRecordCreated

Feature: User trades stocksScenario: User requests a sell before close of trading

Given I have 100 shares of MSFT stockAnd I have 150 shares of APPL stockAnd the time is before close of trading

When I ask to sell 20 shares of MSFT stock

Then I should have 80 shares of MSFT stockAnd I should have 150 shares of APPL stockAnd a sell order for 20 shares of MSFT stock should have been executed

http://martinfowler.com/bliki/GivenWhenThen.html - Pete Hodgson

Given I run a scenario as a LawyerAnd I create a document[And I wait 5 seconds]

When I search the logs APIThen I should find a recent entry for “AuditRecordCreated”

Recap

use logging as a channel/vector to make distributed

systems more testable

unique-ish event type IDs

transaction tracing

log aggregation & search

Stories: expected search results

cross-component visibility

increased test coverage

programmatic assertions

deep insights

trace events in a distributed,

asynchronous system

‘Structured Logging’TW: “Adopt” (May 2015)

https://www.thoughtworks.com/radar/techniques/structured-logging

http://gregoryszorc.com/

.NET: http://serilog.net/Java: https://github.com/fluent/fluent-logger-java

More

Ditch the Debugger and Use Log Analysis Instead

Matthew Skelton

https://blog.logentries.com/2015/07/ditch-the-debugger-and-use-log-analysis-instead/

Thank you

http://skeltonthatcher.com/enquiries@skeltonthatcher.com

@SkeltonThatcher

+44 (0)20 8242 4103

@matthewpskelton