A/B testing with VCS

Guillaume Quintard

What is A/B testing?

● Getting info directly from the user’s behavior

● Change only one element, see how it goes● An excuse to test stuff in prod

How does that work?Access: the user got the content we are testingConversion: the user did what we wanted him to do (buy, submit address,...)

For each group, we compute:conversion rate = conversions/accesses

The group/content with the highest conversion rate wins.

The tools of the trade

● Varnish Cache Plus● Varnish Configuration Language (VCL)● Varnish Custom Statistics (VCS)● Some know-how

Varnish & VCL

● Awesomely fast HTTP Cache● Highly flexible● VCL allows you to inspect http objects and

make decisions/modifications based on this

sub vcl_recv { if (req.url ~ “^/admin/”) {

return (pass) }

}

VCL

Varnish Custom Statistics

● Special purpose in-memory time series database

● Aggregates data about transactions you care about, grouping them by key

● Uses VCL to assign keys to transactions● Plugs into Varnish log

Time windows{ "timestamp": "2015-11-18T14:33:30", "n_req": 434876, "n_miss": 3, "n_bodybytes": 248314196, "reqbytes": 32615703, "respbytes": 409345801, "berespbytes": 2268, "bereqbytes": 566, "ttfb_miss": 0.000231, "ttfb_hit": 0.000028, ... }

Time series

sub vcl_recv {if (req.url ~ “^/admin/”) {

std.log(“vcs-key: AdminAccess”);return (pass)

} }

Tagging transactions

VCS: architecture

Ok, back to A/B testingWe have to:

1. retrieve the group of a user, or select one for new users

2. make that group stick3. deliver different content based on group4. log accesses and conversions5. compute the results6. make a decision7. become rich thanks to that

Introduction togroup theory

sub vcl_recv {if (cookie.get("abgroup") ~ “~[AB]$”) {

set req.http.abgroup =cookie.get("abgroup");

} else { if (std.random(0, 99) < 90) {

set req.http.abgroup = "A";} else {

set req.http.abgroup = "B";}

} }

sub vcl_deliver{set resp.http.Set-Cookie =

"abgroup=" + req.http.abgroup +"; Expires=" +

cookie.format_rfc1123(now, 60m) +"path=/; httpOnly";

}

Come to the dark side,we have cookies

Content selection

Three (main) ways:● url rewriting in Varnish● Backend selection in Varnish● Vary headers

vcl_recv {if (req.url ~ "^/content.jpg") {

if (req.http.abgroup == "A") {set req.url = "/A/content.jpg";

} else {set req.url = "/B/content.jpg";

}}

}

Location rewrite

vcl_recv {if (req.url ~ "^/content.jpg") {

if (req.http.abgroup == "A") {set req.backend_hint = backendA;

} else {set req.backend_hint = backendB;

}}

}

Different backend

Vary headers

No additional VCL!

Vary headers

No additional VCL!Really!

Vary headers

No additional VCL!Really!well...

Vary headers

If the backend respond with a “vary”, the indicated header(s) must match

Vary headers

Vary headers

Vary headers

So, no extra VCL, but we need:● to set the abgroup header (done)● configure the backend to serve different

content depending on that header● configure the backend to reply with

“vary: abgroup”

if (req.url == "/") { std.log("vcs-key: access" +

req.http.abgroup); } else if (req.url == "/thanks.html") { std.log("vcs-key: conversion" +

req.http.abgroup); } }

Log accesses/conversions

Putting it all together

Questions?